Building an Image Annotation Tool — draw shapes, add comments, export JSON

This article shows a complete, practical approach to build an image annotation tool where users can draw shapes (rectangles, ellipses, polygons, freehand), attach comments to shapes, edit/delete annotations, undo/redo, and export/import annotations as JSON. The solution is front-end focused (Angular) with optional ASP.NET Core endpoints to persist annotations. Diagrams and sample code are included so you can implement quickly.

Goals (short)

Load an image and overlay an SVG canvas.
Draw shapes: rectangle, ellipse, polygon, freehand.
Select, move, resize, edit shapes.
Attach comment notes to shapes (popover).
Undo / redo, delete annotation.
Export / import all annotations as JSON (and save via API).
Lightweight — no heavy third-party libraries required.

High-level flowchart

User opens image
       ↓
User chooses tool (rect, ellipse, polygon, freehand, select)
       ↓
User draws on SVG overlay (mouse/touch)
       ↓
Create Annotation object (shape + metadata)
       ↓
User can add/edit comment, move, resize, or delete
       ↓
User exports JSON or saves to server

Workflow (compact)

Load image → Initialize SVG overlay → Choose tool → Draw/Finish shape → Open comment editor → Save annotation (local state)
 → (Optional) Save to server → Export JSON

ER diagram (annotation data model)

+-----------------------+
| Annotation            |
+-----------------------+
| AnnotationId (PK)     |
| ImageId               |
| Type (rect,ellipse,...)|
| ShapeJson (geometry)  |  -- (x,y,w,h) or points array
| CommentText           |
| CreatedBy             |
| CreatedOn             |
| ModifiedOn            |
+-----------------------+

Architecture diagram (Visio-style)

+---------------------+        +---------------------+
|   Angular Frontend  | <----> |   ASP.NET Core API  |
| (Image + SVG Canvas)|        | (save / load JSON)  |
+---------------------+        +---------------------+
         |                              |
         v                              v
   LocalState (annotations)         SQL Server / Blob
   (undo/redo stack)               (persist JSON blobs)

Sequence diagram (draw -> comment -> save)

User -> UI: select rectangle tool
User -> UI: mousedown/drag/mouseup
UI -> State: create annotation object
UI -> UI: open comment input for the annotation
User -> UI: enter comment -> save
UI -> API: POST /annotations (optional)
API -> DB: store JSON
UI -> User: show persisted confirmation

Data model (TypeScript)

Create a simple, extendable model.

src/app/models/annotation.model.ts

export type ShapeType = 'rect' | 'ellipse' | 'polygon' | 'freehand';

export interface ShapeRect {
  x: number; y: number; width: number; height: number;
}
export interface ShapeEllipse {
  cx: number; cy: number; rx: number; ry: number;
}
export interface ShapePolygon {
  points: {x:number,y:number}[];
}
export type ShapeGeometry = ShapeRect | ShapeEllipse | ShapePolygon | { points: {x:number,y:number}[] };

export interface Annotation {
  id: string;                  // guid or uuid
  imageId?: string;            // which image
  type: ShapeType;
  geometry: ShapeGeometry;     // shape-specific geometry
  comment?: string;
  createdBy?: string;
  createdOn?: string;          // ISO datetime
  modifiedOn?: string;
  meta?: any;                  // styling, tags, color, etc.
}

Front-end: core concepts

Image displayed inside a container with an SVG overlay that exactly matches the image size.
All shapes rendered as SVG elements (rect, ellipse, polygon, path).
User interactions controlled by a tool state: select | rect | ellipse | polygon | freehand.
Use normalized coordinates relative to image (0..1) or pixel coordinates — choose one. I recommend percent (0..1) for export portability; sample below uses pixels for simplicity.
Maintain an annotations array + undoStack/redoStack.
Provide simple UI controls for tools and export.

Angular implementation (essential files)

1) Service: annotation-state

src/app/services/annotation.service.ts

import { Injectable } from '@angular/core';
import { BehaviorSubject } from 'rxjs';
import { Annotation } from '../models/annotation.model';
import { v4 as uuid } from 'uuid';

@Injectable({ providedIn: 'root' })
export class AnnotationService {
  private _annotations = new BehaviorSubject<Annotation[]>([]);
  annotations$ = this._annotations.asObservable();

  private undoStack: Annotation[][] = [];
  private redoStack: Annotation[][] = [];

  private pushState() {
    this.undoStack.push(JSON.parse(JSON.stringify(this._annotations.value)));
    this.redoStack = [];
  }

  getAll() { return this._annotations.value; }

  add(a: Partial<Annotation>) {
    this.pushState();
    const ann: Annotation = {
      id: a.id || uuid(),
      type: a.type!,
      geometry: a.geometry!,
      comment: a.comment || '',
      createdOn: new Date().toISOString(),
      ...a
    } as Annotation;
    const cur = [...this._annotations.value, ann];
    this._annotations.next(cur);
    return ann;
  }

  update(id: string, patch: Partial<Annotation>) {
    this.pushState();
    const cur = this._annotations.value.map(x => x.id === id ? {...x, ...patch, modifiedOn: new Date().toISOString()} : x);
    this._annotations.next(cur);
  }

  delete(id: string) {
    this.pushState();
    const cur = this._annotations.value.filter(x => x.id !== id);
    this._annotations.next(cur);
  }

  clear() {
    this.pushState();
    this._annotations.next([]);
  }

  undo() {
    if (!this.undoStack.length) return;
    this.redoStack.push(this._annotations.value);
    this._annotations.next(this.undoStack.pop()!);
  }

  redo() {
    if (!this.redoStack.length) return;
    this.undoStack.push(this._annotations.value);
    this._annotations.next(this.redoStack.pop()!);
  }

  exportJSON() {
    return JSON.stringify(this._annotations.value, null, 2);
  }

  importJSON(json: string) {
    this.pushState();
    const arr: Annotation[] = JSON.parse(json);
    this._annotations.next(arr);
  }
}

Note: uuid package required: npm install uuid --save and import types if needed.

2) Component: image-annotator (core)

src/app/components/image-annotator/image-annotator.component.ts

import { Component, ElementRef, OnInit, ViewChild, HostListener } from '@angular/core';
import { AnnotationService } from '../../services/annotation.service';
import { Annotation } from '../../models/annotation.model';

@Component({
  selector: 'app-image-annotator',
  templateUrl: './image-annotator.component.html',
  styleUrls: ['./image-annotator.component.scss']
})
export class ImageAnnotatorComponent implements OnInit {
  @ViewChild('imageEl') imageEl!: ElementRef<HTMLImageElement>;
  @ViewChild('svgEl') svgEl!: ElementRef<SVGSVGElement>;
  imageSrc = 'assets/sample.jpg'; // your image
  tool: 'select'|'rect'|'ellipse'|'polygon'|'freehand' = 'select';
  drawing = false;
  startX = 0; startY = 0;
  currentId: string | null = null;
  polygonPoints: {x:number,y:number}[] = [];
  freehandPoints: {x:number,y:number}[] = [];

  annotations: Annotation[] = [];
  selectedId?: string;

  constructor(private state: AnnotationService) {}

  ngOnInit() {
    this.state.annotations$.subscribe(a => this.annotations = a);
  }

  // map mouse to SVG coordinates
  getSvgPoint(evt: MouseEvent) {
    const svg = this.svgEl.nativeElement;
    const pt = svg.createSVGPoint();
    pt.x = evt.clientX; pt.y = evt.clientY;
    const ctm = svg.getScreenCTM();
    if (!ctm) return {x:0,y:0};
    const p = pt.matrixTransform(ctm.inverse());
    return { x: p.x, y: p.y };
  }

  onMouseDown(evt: MouseEvent) {
    if (this.tool === 'select') return;
    const p = this.getSvgPoint(evt);
    this.drawing = true;
    this.startX = p.x; this.startY = p.y;

    if (this.tool === 'polygon') {
      // polygon: add point; double click to finish
      this.polygonPoints.push({x:p.x,y:p.y});
    } else if (this.tool === 'freehand') {
      this.freehandPoints = [{x:p.x,y:p.y}];
    } else {
      // create a temporary annotation id for rect/ellipse
      const ann = this.state.add({
        type: this.tool,
        geometry: (this.tool === 'rect') ? {x:p.x,y:p.y,width:0,height:0} : {cx:p.x,cy:p.y,rx:0,ry:0}
      });
      this.currentId = ann.id;
    }
  }

  onMouseMove(evt: MouseEvent) {
    if (!this.drawing) return;
    const p = this.getSvgPoint(evt);

    if (this.tool === 'freehand') {
      this.freehandPoints.push({x:p.x,y:p.y});
      // update current freehand annotation
      if (!this.currentId) {
        const ann = this.state.add({ type: 'freehand', geometry: { points: this.freehandPoints }});
        this.currentId = ann.id;
      } else {
        this.state.update(this.currentId, { geometry: { points: this.freehandPoints }});
      }
      return;
    }

    if (this.currentId && (this.tool === 'rect' || this.tool === 'ellipse')) {
      const dx = p.x - this.startX;
      const dy = p.y - this.startY;
      if (this.tool === 'rect') {
        const rect = { x: Math.min(this.startX,p.x), y: Math.min(this.startY,p.y),
                       width: Math.abs(dx), height: Math.abs(dy) };
        this.state.update(this.currentId, { geometry: rect });
      } else if (this.tool === 'ellipse') {
        const ellipse = { cx: (this.startX + p.x)/2, cy: (this.startY+p.y)/2, rx: Math.abs(dx/2), ry: Math.abs(dy/2) };
        this.state.update(this.currentId, { geometry: ellipse });
      }
    }
  }

  onMouseUp(evt: MouseEvent) {
    if (!this.drawing) return;
    this.drawing = false;

    if (this.tool === 'polygon') {
      // polygons finish on double-click (handled elsewhere) — here do nothing
      return;
    }

    if (this.tool === 'freehand') {
      this.currentId = null;
      return;
    }

    // for rect/ellipse finalize and open comment editor (optionally)
    if (this.currentId) {
      // open comment input UI — maybe select and show a small popup
      this.selectedId = this.currentId;
      this.currentId = null;
    }
  }

  finishPolygon() {
    if (this.polygonPoints.length < 3) return;
    const ann = this.state.add({ type: 'polygon', geometry: { points: this.polygonPoints }});
    this.polygonPoints = [];
    this.selectedId = ann.id;
  }

  selectAnnotation(id?: string) { this.selectedId = id; }

  updateComment(id: string, comment: string) {
    this.state.update(id, { comment });
  }

  deleteAnnotation(id: string) { this.state.delete(id); }

  undo() { this.state.undo(); }
  redo() { this.state.redo(); }

  exportAnnotations() {
    const json = this.state.exportJSON();
    // download
    const blob = new Blob([json], { type: 'application/json' });
    const url = URL.createObjectURL(blob);
    const a = document.createElement('a'); a.href = url; a.download = 'annotations.json'; a.click();
    URL.revokeObjectURL(url);
  }

  importAnnotations(file: File) {
    const reader = new FileReader();
    reader.onload = () => { this.state.importJSON(String(reader.result)); };
    reader.readAsText(file);
  }
}

3) Component template (SVG overlay + UI)

src/app/components/image-annotator/image-annotator.component.html

<div class="toolbar">
  <button (click)="tool='select'">Select</button>
  <button (click)="tool='rect'">Rect</button>
  <button (click)="tool='ellipse'">Ellipse</button>
  <button (click)="tool='polygon'">Polygon</button>
  <button (click)="tool='freehand'">Freehand</button>
  <button (click)="undo()">Undo</button>
  <button (click)="redo()">Redo</button>
  <button (click)="exportAnnotations()">Export JSON</button>
  <input type="file" (change)="importAnnotations($event.target.files[0])"/>
</div>

<div class="canvas-wrap">
  <img #imageEl [src]="imageSrc" class="base-image" alt="annotate image"/>

  <svg #svgEl class="overlay"
       (mousedown)="onMouseDown($event)"
       (mousemove)="onMouseMove($event)"
       (mouseup)="onMouseUp($event)">

    <!-- render existing annotations -->
    <g *ngFor="let a of annotations">
      <rect *ngIf="a.type==='rect'"
            [attr.x]="(a.geometry as any).x"
            [attr.y]="(a.geometry as any).y"
            [attr.width]="(a.geometry as any).width"
            [attr.height]="(a.geometry as any).height"
            [attr.stroke]="selectedId === a.id ? 'red' : 'yellow'"
            stroke-width="2" fill="transparent"
            (click)="selectAnnotation(a.id)"/>
      <ellipse *ngIf="a.type==='ellipse'"
            [attr.cx]="(a.geometry as any).cx"
            [attr.cy]="(a.geometry as any).cy"
            [attr.rx]="(a.geometry as any).rx"
            [attr.ry]="(a.geometry as any).ry"
            [attr.stroke]="selectedId === a.id ? 'red' : 'cyan'"
            stroke-width="2" fill="transparent"
            (click)="selectAnnotation(a.id)"/>
      <polygon *ngIf="a.type==='polygon'"
            [attr.points]="(a.geometry as any).points.map(p=>p.x+','+p.y).join(' ')"
            [attr.stroke]="selectedId === a.id ? 'red' : 'lime'"
            stroke-width="2" fill="transparent"
            (click)="selectAnnotation(a.id)"/>
      <polyline *ngIf="a.type==='freehand'"
            [attr.points]="(a.geometry as any).points.map(p=>p.x+','+p.y).join(' ')"
            [attr.stroke]="selectedId === a.id ? 'red' : 'orange'"
            stroke-width="2" fill="none"
            (click)="selectAnnotation(a.id)"/>
    </g>
  </svg>

  <!-- comment editor / inspector -->
  <div class="inspector" *ngIf="selectedId">
    <div *ngFor="let a of annotations" >
      <div *ngIf="a.id === selectedId">
        <h4>Annotation</h4>
        <label>Type: {{a.type}}</label>
        <textarea [(ngModel)]="a.comment" (blur)="updateComment(a.id, a.comment)"></textarea>
        <button (click)="deleteAnnotation(a.id)">Delete</button>
      </div>
    </div>
  </div>
</div>

<!-- polygon finish button -->
<div *ngIf="tool==='polygon'">
  <button (click)="finishPolygon()">Finish Polygon</button>
</div>

4) Component styles (basic)

src/app/components/image-annotator/image-annotator.component.scss

.canvas-wrap { position: relative; display:inline-block; }
.base-image { display:block; max-width:100%; }
.overlay {
  position: absolute;
  top: 0; left: 0;
  width: 100%;
  height: 100%;
  pointer-events: all;
}
.overlay rect, .overlay ellipse, .overlay polygon { cursor: pointer; }
.toolbar { margin-bottom: 8px; }
.inspector { position: absolute; right: 8px; top: 8px; background: rgba(255,255,255,0.9); padding: 8px; border-radius:4px; }

Important: Ensure the SVG overlay is sized to the displayed image. You may set the SVG viewBox to image natural width/height and make overlay preserveAspectRatio to match image scaling. For simplicity above the overlay uses absolute positioning and assumes image shows at natural size; for responsive layouts compute scale factors and transform coordinates accordingly.

Backend: Persisting annotations (ASP.NET Core sample)

You can persist annotation JSON as a blob in a database table.

DB schema (simple)

CREATE TABLE ImageAnnotations (
  Id INT IDENTITY PRIMARY KEY,
  ImageId NVARCHAR(200),
  AnnotationJson NVARCHAR(MAX),
  CreatedBy NVARCHAR(200),
  CreatedOn DATETIME2 DEFAULT SYSUTCDATETIME()
);

Controller (C#)

AnnotationController.cs

[ApiController]
[Route("api/[controller]")]
public class AnnotationController : ControllerBase
{
    private readonly IConfiguration _cfg;
    public AnnotationController(IConfiguration cfg) { _cfg = cfg; }

    [HttpPost("save")]
    public async Task<IActionResult> Save([FromBody] SaveDto dto)
    {
        var json = dto.AnnotationJson;
        using var conn = new SqlConnection(_cfg.GetConnectionString("DefaultConnection"));
        using var cmd = new SqlCommand("INSERT INTO ImageAnnotations (ImageId, AnnotationJson, CreatedBy) VALUES (@img, @json, @by)", conn);
        cmd.Parameters.AddWithValue("@img", dto.ImageId);
        cmd.Parameters.AddWithValue("@json", json);
        cmd.Parameters.AddWithValue("@by", dto.CreatedBy ?? "system");
        conn.Open();
        await cmd.ExecuteNonQueryAsync();
        return Ok();
    }

    [HttpGet("load/{imageId}")]
    public async Task<IActionResult> Load(string imageId)
    {
        using var conn = new SqlConnection(_cfg.GetConnectionString("DefaultConnection"));
        using var cmd = new SqlCommand("SELECT TOP 1 AnnotationJson FROM ImageAnnotations WHERE ImageId=@img ORDER BY CreatedOn DESC", conn);
        cmd.Parameters.AddWithValue("@img", imageId);
        conn.Open();
        var res = await cmd.ExecuteScalarAsync();
        return Ok(new { json = res });
    }
}

public class SaveDto { public string ImageId { get; set; } = ""; public string AnnotationJson { get; set; } = ""; public string? CreatedBy { get; set; } }

UX details & improvements (practical tips)

Coordinate normalization: store coordinates normalized to the image natural width/height (0..1). On rendering multiply by current display width/height. This avoids scaling issues on different displays.
Hit testing & selection: use elementFromPoint or reverse-map by math; for polygons implement point-in-polygon algorithm to detect clicks.
Resize handles: add small draggable squares on corners for rect and ellipse. Manage mouse/touch events for resizing.
Undo/redo: service above stores full snapshot; for large annotation sets consider storing deltas.
Performance: for many annotations prefer canvas rather than SVG (but SVG is easier for selection/edit).
Freehand smoothing: use polyline simplification (Ramer–Douglas–Peucker) for cleaner export.
Comment popover: position comment box near shape centroid; use absolute positioned div overlay.
Export format: include imageId, version, author, timestamp, shapes array with type, geometry, comment, color, id.

Example export JSON:

{
  "imageId":"img-123",
  "exportedOn":"2025-11-17T10:00Z",
  "annotations":[
    {"id":"a1","type":"rect","geometry":{"x":10,"y":20,"width":100,"height":60},"comment":"damage here","createdOn":"..."},
    {"id":"a2","type":"polygon","geometry":{"points":[{"x":100,"y":50},{"x":120,"y":80},{"x":90,"y":95}]},"comment":"crack"}
  ]
}

Accessibility & Mobile

Support touch events (touchstart, touchmove, touchend) for drawing on mobile. Map touch coordinates to SVG similar to mouse.
Provide keyboard shortcuts for tools and undo/redo.
Ensure color contrast for annotation strokes and comment boxes.
Allow zoom & pan of the image — when panning, keep overlay synchronized.

Testing checklist

Draw every shape type and export/import — ensure geometry round-trip.
Resize and move shapes and verify updated geometry.
Undo/redo sequences (multiple steps) — verify state integrity.
Large images & different DPI — verify scaling.
Multi-touch interactions on mobile.
Save/load via API; simulate concurrent saves.