跳转至

Memory Transfer Mode Design

Design notes for the TWAIN Memory Transfer (TWSX_MEMORY) path in BN Tech Virtual Scanner.

1. Requirement

The virtual scanner must support TWAIN's Memory Transfer mode (ICAP_XFERMECH = TWSX_MEMORY): instead of returning the entire image as one DIB handle, the DS copies the image strip-by-strip into a buffer the application pre-allocates, until the whole image has been delivered.

Main requirements:

  • ICAP_XFERMECH must add TWSX_MEMORY to its supported list ({TWSX_NATIVE, TWSX_FILE, TWSX_MEMORY}); the default remains TWSX_NATIVE.
  • Support DG_CONTROL / DAT_SETUPMEMXFER / MSG_GET, returning MinBufSize / MaxBufSize / Preferred.
  • Support DG_IMAGE / DAT_IMAGEMEMXFER / MSG_GET, copying whole-scanline chunks into the app-supplied TW_MEMORY buffer and filling TW_IMAGEMEMXFER.
  • Pixel byte order must follow TWAIN spec: TWPT_RGB must be delivered as R-G-B, not Windows DIB's B-G-R.
  • Row order must be top-down: first strip contains the topmost rows; YOffset increases monotonically.
  • Compression = TWCP_NONE (raw uncompressed pixels).
  • Return TWRC_XFERDONE on the last strip, TWRC_SUCCESS otherwise.
  • State machine: kEnabled (5) -> kXferReady (6) -> kXferring (7) -> kEnabled (5), identical to Native / File.
  • Reuse the existing VirtualScanner and transfer() pipeline with minimal code changes.

Non-functional:

  • Share acquireImage + preScanPrep + getScanStrip with Native / File.
  • Identical behavior on 32-bit and 64-bit DS.
  • Per-strip size is chosen by the application; the DS must tolerate any buffer (typically 4 KB – 1 MB) and silently round down to whole rows.
  • Large images (A4 600 DPI 24-bit ≈ 100 MB) must transfer correctly with acceptable R/B-swap overhead.

2. Domain knowledge

2.1 Protocol sequence

1. MSG_ENABLEDS -> acquireImage(), MSG_XFERREADY (5 -> 6)
2. DAT_EVENT / MSG_PROCESSEVENT
3. DAT_IMAGEINFO / MSG_GET
4. DAT_SETUPMEMXFER / MSG_GET -> {Min, Max, Preferred}
5. App allocates a TW_MEMORY buffer
6. Loop:
   DAT_IMAGEMEMXFER / MSG_GET  (TheMem allocated)
     -> middle strips: TWRC_SUCCESS
     -> last strip:    TWRC_XFERDONE
     state 6 -> 7 on the first call
7. DAT_PENDINGXFERS / MSG_ENDXFER (7 -> 5)
8. MSG_DISABLEDS (5 -> 4)

Return codes: TWRC_SUCCESS, TWRC_XFERDONE, TWRC_CANCEL, TWCC_SEQERROR, TWCC_BADVALUE.

2.2 TW_SETUPMEMXFER

MinBufSize / MaxBufSize / Preferred. Apps usually allocate Preferred; the DS must tolerate any value in [MinBufSize, MaxBufSize].

2.3 TW_IMAGEMEMXFER

Compression, BytesPerRow, Columns, Rows, XOffset, YOffset, BytesWritten, Memory{Flags, Length, TheMem}. Memory.TheMem is the app-allocated destination.

2.4 Row and byte order

  • Row order: TWAIN Memory mode is top-down (opposite of Windows DIB). First strip = topmost rows; YOffset grows from 0.
  • Byte order: TWPT_RGB requires R-G-B in Memory mode; FreeImage / Windows DIB uses B-G-R. 24-bit data must be R/B-swapped on the way out.
  • Row alignment: 4-byte padded, matching DIB convention (BytesPerRow = ((Columns * BPP) + 31) / 32 * 4).

2.5 Strip slicing

  • Each DAT_IMAGEMEMXFER call writes only whole rows: rows = min(app_buf, remaining) / BytesPerRow.
  • App buffer must hold at least one row; otherwise the DS returns TWRC_FAILURE + TWCC_BADVALUE.
  • A single row is never split across two calls.

2.6 Relation to Native Transfer

Native already materializes the whole image into image_data_ via transfer(). Memory Transfer can reuse that buffer and just slice it.

3. Design goals

  • 100% compatibility with TWAIN 2.x Memory Transfer apps (Twack 32, NAPS2, Dynamic Web TWAIN, …).
  • Reuse transfer() for image materialization; avoid duplicating the pixel pipeline.
  • Minimal diff: 2 new handlers, 1 new offset member, 1-line capability change; no touch to VirtualScanner or settings UI.
  • Single output point for the protocol-specific byte order: only the Memory path swaps R/B.

Non-goals: compression (Compression = TWCP_NONE only), streaming on-demand decoding, > 8-bit-per-sample pixel types, dynamic Min/Max/Preferred, exposing strip size to the UI.

4. Overall design

TwainDataSource
├── handleDatSetupMemXfer()       // returns 8K / 256K / 64K
├── handleDatImageMemXfer()       // strip slicer
│     ├── (auto-promote 5 -> 6)
│     ├── transfer() on first call -> image_data_
│     ├── slice rows = min(app_buf, remaining) / BytesPerRow
│     ├── memcpy + 24-bit R/B swap
│     └── return TWRC_SUCCESS / TWRC_XFERDONE
└── reused: transfer / getImageInfo / endXfer / resetXfer / DSM mem shim

Data flow:

FreeImage DIB (BGR, bottom-up internally)
        │
        ▼ getScanStrip() flips index -> visually top-down
image_data_  (BGR, top-down, 4-byte aligned)
        │
        ├──> Native: copyDibPixelData() flips rows again -> DIB bottom-up (BGR ok)
        └──> Memory: memcpy rows -> in-place R/B swap -> app buffer (RGB, top-down)

5. Key decisions and rationale

5.1 Reuse transfer() and slice image_data_

Minimal code; predictable behavior; image_info stays accurate. Cost: same memory footprint as Native, which is acceptable since Memory mode usually serves "process strips as they arrive" rather than DS-side memory savings.

5.2 Fixed MinBufSize / Preferred / MaxBufSize

8 KB / 64 KB / 256 KB. Simple, predictable, can answer even before image_info_ is ready.

5.3 R/B swap only for 24-bit

TWAIN spec requires R-G-B for TWPT_RGB; FreeImage gives B-G-R on Windows. 8-bit / 1-bit have no channels to swap. Swap is done per-strip, not on the whole buffer, so amortized cost is small.

5.4 Reject when app buffer < one row

Preserves the Rows * BytesPerRow == BytesWritten invariant that apps assume. Apps recover by reallocating to Preferred.

5.5 Reuse the state 5 -> 6 auto-promote shim

Same compatibility shim as Native. Twack and similar test apps occasionally skip MSG_PROCESSEVENT.

5.6 Single mem_xfer_offset_ as the only strip state

Minimal state; YOffset = mem_xfer_offset_ / BytesPerRow; reset in three centralized places (ctor / endXfer / resetXfer).

5.7 No compression

Out of scope for the minimum-diff goal; Memory-mode customers usually want raw pixels for downstream processing.

5.8 Byte-order conversion lives at the protocol exit, not in image_data_

Single responsibility: protocol-specific conventions stay at the protocol output. Keeps Native / File paths untouched.

6. Component changes

6.1 src/capability.cpp

  • Append TWSX_MEMORY to ICAP_XFERMECH supported values.

6.2 src/twain_data_source.h

  • Declare handleDatSetupMemXfer and handleDatImageMemXfer.
  • Add TW_UINT32 mem_xfer_offset_ member.

6.3 src/twain_data_source.cpp

  • Initialize mem_xfer_offset_(0) in constructor.
  • Add dispatches in DG_CONTROL / DG_IMAGE switches.
  • Reset mem_xfer_offset_ in endXfer / resetXfer.
  • Implement the two handlers (see ​§4 / §6.3 in the Chinese section).

6.4 Untouched

VirtualScanner, settings_server.cpp, ds_entry.cpp, DSM memory shim.

6.5 Build

No new files, no new dependencies. Both 32-bit and 64-bit builds verified.

7. Typical flows

7.1 Twack 32 Memory Transfer round trip

1. MSG_OPENDS
2. ICAP_XFERMECH / MSG_SET = TWSX_MEMORY
3. MSG_ENABLEDS, ShowUI=FALSE -> MSG_XFERREADY
4. DAT_IMAGEINFO -> 2480x3508, BPP=24
5. DAT_SETUPMEMXFER -> {8K, 256K, 64K}
6. Allocate 64 KB
7. Loop DAT_IMAGEMEMXFER:
   - first call: transfer() materializes image_data_; state 6 -> 7
   - rows = 64 KB / 7440 ≈ 8 rows per strip; R/B swap; advance offset
   - last strip: TWRC_XFERDONE
8. MSG_ENDXFER -> MSG_DISABLEDS

7.2 Grayscale Memory Transfer

ICAP_PIXELTYPE = TWPT_GRAY
ICAP_XFERMECH = TWSX_MEMORY
DAT_IMAGEMEMXFER: 8-bit, no R/B swap, just memcpy rows.

7.3 Buffer-too-small recovery

App buffer = 100 bytes, BytesPerRow = 7440
-> TWRC_FAILURE + TWCC_BADVALUE
App reallocates to 64 KB and retries successfully.

8. Limitations

  • Whole image materialized into image_data_ (no true streaming savings on DS side).
  • No compression (TWCP_NONE only).
  • No high-bit-depth pixel types.
  • Fixed Min/Max/Preferred, not adaptive to resolution or pixel type.
  • mem_xfer_offset_ is monotonic; no strip resend.
  • Auto-promote 5 -> 6 is a compatibility shim, not strictly spec-compliant.
  • No progress callback for the settings UI.
  • 24-bit R/B swap is scalar in-place; not SIMD-optimized yet.
  • Memory.Flags is ignored; the buffer is always treated as a plain pointer.

9. Next steps

  • Validate more apps: NAPS2, Dynamic Web TWAIN, ScandAll PRO, ImageGear.
  • Implement compressed strips (TWCP_GROUP4 for BW, TWCP_JPEG for color).
  • Add 16 / 48-bit-per-sample support.
  • Make Min/Preferred/Max adaptive (e.g. Preferred = 4 * BytesPerRow).
  • Add a strip-copy progress callback for the settings UI.
  • Introduce a true streaming pipeline so image_data_ is not held entirely in memory.
  • SIMD-optimize the 24-bit R/B swap; or emit RGB directly from transfer() when the active mech is Memory.
  • Expose "default transfer mode" in the settings UI for easier manual testing.
  • Add automated tests: stub DSM + bit-exact comparison between Native and Memory outputs.
  • Investigate additional memory request modes (e.g. TWMR_INVERT, TWMR_DUAL); current implementation supports single-buffer loop only.