Commit Graph
18 Commits
Author SHA1 Message Date
Charles YaoandClaude Opus 5 cf99766d93 fix(dify-builder): count nodes the chokepoint normalizers healed as changed
apply_repair's value normalizers can heal a node the repair's intents
never touched; the diff showed it but changed_nodes did not. Both
normalizers' changed ids now join changed_nodes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 00:43:35 -07:00
Charles YaoandClaude Opus 5 d299a81a9f fix(dify-builder): normalize values at the write chokepoint; reject undeclared handles
apply_repair runs the same graph_normalizers the generator's postprocess
runs, for the intents the generator never saw: a Fix/Edit set_node_config
that writes a condition value as a JSON number (ESQ1-285's flip-flop) or
an http body item without type is healed before the preflight would
reject it.

apply_connect rejects an edge from a branch node on a handle the node
does not declare -- including the default "source" it used to fill in
-- with the same ValueError the handlers already surface as "Couldn't
apply the fix". Such an edge hangs off nothing and its arm never runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 00:27:44 -07:00
Charles YaoandClaude Opus 5 fcb7c719ef feat(dify-builder): preflight the draft in apply_repair before writing it
ESQ1-302 and ESQ1-303 both died at Graph.init -- before workflow_started,
so the Builder saw an error frame with no run id and no rows, and spent
runs and approvals discovering what a validation pass could have said
up front. apply_repair now runs that pass (preflight_errors: the same
node-data validation create_node performs, with no factory or runtime
state) over the graph it is about to write and raises PreflightError,
naming each node Graph.init would reject.

Only NEW problems count: a pre-existing broken node the repair does not
touch must not veto an unrelated fix, and a repair that heals it passes.
PreflightError is a ValueError so the handlers' existing stale-intent
guards already surface it as a card.

The real ESQ1-303 dev draft is the fixture: preflight names node2 and
its numeric value, and passes once the value is a string.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 21:21:17 -07:00
Charles YaoandClaude Opus 5 f862fe327e fix(dify-builder): a stream that ends on an error frame is a failed run
run_draft forwarded the pipeline's ``event: error`` frame to the frontend
and then dropped it, so the ESQ1-302 launch failure (Graph.init rejected
an http-request node before workflow_started) fell into the
truncated-stream path and was reported as "running". The handler's
unknown-outcome branch then sent the user back to re-run, forever.

The loop now keeps the first error frame aside and _finish_run applies
one precedence: terminal frame > error frame > nothing. Chatflow emits
workflow_finished(failed) AND an error frame for one failure, so the
terminal frame must win; a stream that ends with neither still reads as
an unknown outcome, never as a fabricated failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 20:51:24 -07:00
twwu d5ca5ea8ef fix(dify-builder): stream native test runs after canvas refresh
Run workflow streaming inside the Builder worker to avoid queue deadlocks. Forward native events to shared editor callbacks and wait for graph refresh before retesting repairs.

Keep command cleanup stable, preserve batched canvas events, and allow failed or unfinished historical runs to refresh again.
2026-09-21 15:12:22 +08:00
Charles YaoandClaude Opus 5 d2e204f133 fix(dify-builder): stop the test step deadlocking and losing its outcome
Two defects, both introduced by the streaming switch in 3838db8aa7, which
together livelocked every Builder test run.

1. Self-deadlock. run_draft executes inside the Celery task
   dify_builder_advance_task.advance_session. Streaming mode does not run
   the workflow in place: it enqueues workflow_based_app_execution_task
   via delay() on first subscribe (app_generate_service.py:322-324) and
   returns a Redis subscription. The advance task then blocked on events
   from a task needing the worker slot the advance task was holding.
   Observed live: advance_session "succeeded in 300.59s" (the idle
   timeout) four times over, while each workflow_run finished in under 2s.
   Deadlocks every time at worker concurrency 1, and as soon as N test
   runs are in flight at concurrency N.

2. Lost outcome. Blocking mode returns a Mapping
   {"task_id", "workflow_run_id", "data"}, but the rewrite consumed the
   response with `for chunk in response`, which walks a dict's KEYS as
   strings. Probed against the real service: 3 chunks, 0 run-id matches,
   0 terminal matches. So every run was recorded status="running" with an
   empty dify_run_id, and the "running" branch bounced the user back to
   build.execution -- forever. The pre-3838db8aa7 code read
   response["data"] directly; the rewrite dropped that path.

Blocking mode restored, Mapping responses handled, and the shared tail
factored into _finish_run so the two paths cannot disagree about an
outcome. The stream-consumption path is kept and still tested, for a
future consumer that is NOT a worker -- which is what live per-node
streaming to the frontend will require.

Verified end to end on a local stack: one test run, status succeeded with
a real run id, advance_session 0.04-31.8s, and the session reaching
build.review instead of looping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-20 22:10:28 -07:00
Wu Tianwei bf8333bc77 fix(builder): avoid layout conflicts and serialize draft updates (#42601) 2026-09-21 09:48:02 +08:00
Charles YaoandClaude Opus 5 574c9d1208 fix(dify-builder): never report a truncated test run as a failed one
Streaming introduced a correctness regression: if the event stream ended
without a terminal frame (idle timeout, dead publishing worker), run_draft
synthesised status="failed" with an empty dify_run_id. A run that had in fact
SUCCEEDED would then be reported as a failed build and sent into the repair
loop. The blocking path would have waited and returned the truth.

The stream is the fast path, not the authority. Every non-ping frame is
stamped with workflow_run_id by convert_stream_full_response, so the run id
is known from the first frame onwards -- long before the terminal frame that
may never arrive. run_draft now captures it from any frame and, when no
terminal frame arrives, reads the run's real status from the workflow-run row
instead of inventing one.

Note that data["id"] is the node-execution id on node frames, not the run id,
so it is only read as a run id on workflow-level frames.

When even the row cannot settle it -- no run id, no row, or a row still
marked running because the worker died mid-run -- the result is reported as
status="running" with an explicit Run.error, which is what Run.status already
documents for "not over / not known". A truncated stream is now either the
truth or a visible unknown, never a fabricated failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-20 15:49:50 -07:00
Charles YaoandClaude Opus 5 e9599355a2 feat(dify-builder): stream the test run instead of replaying it
run_draft blocked on AppGenerateService.generate(streaming=False) and then
replayed every finished node-execution row through on_event, so the user
watched a spinner for the whole run and then saw every node light up at once.
It now consumes the streaming generator and fires on_event per frame.

Two things the plan assumed that turned out not to hold:

- The stream does NOT yield dicts. convert_stream_full_response yields dicts,
  but that runs in the Celery worker; on our side
  BaseAppGenerator.convert_to_event_stream (not overridden) re-wraps every
  event as "data: {json}\n\n" and every keep-alive as "event: ping\n\n". An
  isinstance(chunk, dict) guard would have dropped every frame. New
  stream_chunk_as_mapping accepts both shapes and rejects keep-alives.

- The streaming workflow_paused frame carries workflow_run_id where the
  blocking response carried id, so map_run_result would KeyError on it.
  Adapted at the call site via run_result_data_from_terminal_chunk;
  map_run_result's contract is unchanged for the fix/edit flows.

Session lifetime: _dispatch_generate is a plain function, so all of its
Session work is eager, and the generator it returns only reads a Redis
subscription while the run executes in a Celery worker with its own session.
The consumption loop therefore sits outside the `with` block rather than
pinning a DB connection for the length of a workflow run; a test pins this.

The node-execution read stays at the end -- map_run_result still needs those
rows for per_node[].outputs. Only the progress streams.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-20 15:42:25 -07:00
twwu c51e277389 feat(dify-builder): show affected nodes in change cards
Render change sets directly in plan-style cards with affected node names,
IDs, and change details. Capture node metadata for proposed and applied
changes and persist it with conversation history.

Remove the view_changes action and expand/collapse state. Update generated
contracts, translations, and regression coverage.

Validation: 112 frontend tests and 742 backend unit tests passed.
2026-09-07 13:44:39 +08:00
Charles YaoandClaude Opus 4.8 31a972e961 Merge PR #41778 (feat/add-progress-event) into feat/dify-builder
Integrates the FE team's live-progress + conversation-streaming rework
(durable progress events, repository layer, snapshot-consistent reads,
idempotent message retries, EDIT_PUBLISH->EDIT_COMPLETE) and reconciles its
interrupted-step recovery model with the reconciler already on
feat/dify-builder.

Combined recovery model:
- interrupted (crash mid working-step, lock released, nothing durably
  recorded) -> Retry (recovery_continue, re-runs the handler) + Start over
- FAILED (handler raised, recorded by runner.fail) -> Restart only
- a Retry that itself raises lands in FAILED -> Restart (no wedge)

Kept from the PR: EDIT_PUBLISH working / EDIT_COMPLETE terminal, snapshot
refresh in apply_recovery_action, paused reset. Kept from the reconciler:
Retry on interrupted, state-aware access tier (EDIT_PUBLISH added to the
publish tier), apply_repair idempotency filter.

680 dify_builder unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-04 02:14:40 -07:00
Charles YaoandClaude Opus 4.8 3dcb06db01 fix(dify-builder): exclude in-batch delete_node ids from apply_repair idempotency filter
apply_repair's already-present filter (added for interrupted-step Retry
idempotency) compared create_node/connect intents against before_graph
alone, with no awareness of delete_node intents in the same batch. A
from-scratch build sends delete_node(placeholder_start) + create_node(same
id) in one apply_repair call to replace the canvas's default placeholder
start with the generator's own start node -- the filter dropped the
create_node as "already present" while the delete_node still ran, deleting
the start node with no re-create, so the following connect raised
ValueError: node not found. Mirror handlers_build.py's existing M2 guard:
subtract in-batch delete_node targets from the present-node/edge sets
before filtering, so a delete+recreate of the same id in one batch survives.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-04 00:01:42 -07:00
Charles YaoandClaude Opus 4.8 61b35a36b7 fix(dify_builder): make apply_repair idempotent for already-present create/connect
Drops create_node intents whose id already exists and connect intents whose
edge already exists (referenced against the original draft graph), so a
fix.apply Retry re-applying the same intents is a clean no-op instead of a
duplicate/ValueError. graph_ops validation (raise on a genuinely fresh
duplicate id) is preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-04 00:01:42 -07:00
twwu eb25c2d8ee feat(dify-builder): add live progress and conversation streaming 2026-09-04 10:16:50 +08:00
Charles YaoandClaude Opus 4.8 1eb5bb1dd1 fix(dify_builder): publish adapter unpacked publish_workflow's return as a tuple
WorkflowService.publish_workflow returns a single Workflow (-> Workflow), but
dify_port.publish still unpacked it as (workflow, retirement_candidates) -- a
stale signature. Every other caller was updated when the return type changed
(via the main merge); only the Dify Builder port was missed, so every publish
crashed with "TypeError: cannot unpack non-iterable Workflow object", surfaced
to the user as a generic "step failed" SSE frame.

Assign the single return value (retirement_candidates was unused anyway). The
existing test_dify_port publish test mocked publish_workflow as a 2-tuple, which
masked the bug -- corrected it to the real -> Workflow signature so it now guards
the regression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-02 16:11:49 -07:00
Charles YaoandClaude Opus 4.8 6714e5a1d0 fix(dify-builder): consolidate APPLY_FNS, simplify filter_applicable, add dup-id test
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-29 01:40:29 -07:00
Charles YaoandClaude Opus 4.8 24de2c8417 Dedup Dify Builder handlers/services: extract shared helpers
Behavior-preserving code-quality cleanup across the Dify Builder engine:
extract emit_canvas, build_change_set, mint_checkpoint, action_kind, and
merge_known_keys into handlers_fix.py (shared by handlers_build.py and
handlers_edit.py); extract _add in repository.py for the id-preservation
idiom; extract _make_edge (and unify the node/edge id-collision loops
into _next_unique_id) in graph_ops.py; extract _sync_graph_only in
dify_port.py. No behavior, generated ids, or wire output changed — same
387-test suite passes, ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-23 19:32:57 -07:00
Charles YaoandClaude Opus 4.8 7e83264330 refactor: rename Workflow Copilot → Dify Builder
Project rename across api/ + web/: modules, symbols, routes
(/console/api/dify-builder/*), feature flag (DIFY_BUILDER_ENABLED),
Celery queue/tasks, the 7 DB tables (via migration df1b0114de40),
and regenerated contract artifacts. Behavior unchanged; 430 tests green,
ruff clean. Branch renamed feat/copilot-py-backend → dify-builder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-23 11:18:35 -07:00