Pages and saved cursors
Storage.list/3 still returns all entries and Storage.stream/3 still emits
entries lazily, raising on failures. For explicit checkpoints use
Storage.list_page/3 and Storage.continue_list/3. They fetch one page and return
{:ok, %Magpie.ListPage{entries: entries, cursor: cursor, has_more: boolean}}
or {:error, exception}. No cursor is stored or acknowledged by Magpie.
client = Magpie.Client.new("ACCESS_TOKEN")
{:ok, first} =
Magpie.Storage.list_page(client, "", recursive: true, include_deleted: true, limit: 100)
Enum.each(first.entries, &IO.inspect/1)
# Only after processing every entry successfully:
saved_cursor = first.cursor
{:ok, next} = Magpie.Storage.continue_list(client, saved_cursor)
Enum.each(next.entries, &IO.inspect/1)
# Repeat while next.has_more; retain the final cursor for future changes.list_page takes the listing keywords documented in Storage.stream/3:
recursive, include_media_info, include_deleted,
include_has_explicit_shared_members, include_mounted_folders, limit
(1–2000, a server hint), shared_link, include_property_groups and
include_non_downloadable_files. Defaults remain Dropbox's defaults.
Both new calls accept request: [...] for client/request overrides; continuation
accepts no listing flags because they are encoded in the cursor. Unsupported
options and empty/non-binary cursors raise ArgumentError before I/O.
Keep cursors with their account, namespace, path and original listing settings.
They are opaque strings, not timestamps. Even an empty terminal page has a new
checkpoint to save. A final cursor can later return more entries. The
Dropbox change guide
explains cursor polling and expiry. API errors retain Magpie.Error, expected
transport failures retain Req exception types, and execution budgets return
Magpie.TimeoutError. Each call has its own execution budget; set an overall
worker deadline in your application if necessary.
Entries use the existing FileMetadata, FolderMetadata and DeletedMetadata
types. Apply file/folder state at the reported path; deletion removes the path
and descendants. Unknown future metadata tags remain maps. Do not infer an event
kind (create, edit, move) from one entry. This is state reconciliation, with no
complete event history or exactly-once delivery guarantee.
Starting with future changes only
If existing entries are intentionally out of scope, request the latest cursor
with the same listing options instead of fetching and discarding every initial
page. This endpoint accepts the Dropbox ListFolderArg map, just like
list_folder/3; its options use string keys, unlike the Storage keywords.
{:ok, %{"cursor" => cursor}} =
Magpie.Files.ListFolder.get_latest_cursor(client, "", %{
"recursive" => true,
"include_deleted" => true
})
# Persist this starting cursor externally; no initial entries are returned.
# Later, when polling or processing a webhook:
{:ok, page} = Magpie.Storage.continue_list(client, cursor)
# Apply page.entries before committing page.cursor; drain while page.has_more.This deliberately skips the current contents. It does not create a local
snapshot and cannot replace full-listing recovery after a reset. Keep the cursor
with its account, namespace and options. get_latest_cursor(client, path) keeps
its original request and {:ok, %{"cursor" => cursor}} result.
Scanner with external checkpoint storage
The following module is compiled directly from this guide by
test/incremental_guide_test.exs. commit_page is provided by your application:
it must apply the page and save its cursor atomically, returning :ok only once
both are durable. For nontransactional side effects use idempotency or an outbox.
The scanner holds only the current page and stops immediately on commit failure.
defmodule MagpieGuide.Scanner do
alias Magpie.Storage
def scan(client, root, saved_cursor, commit_page) do
result =
if saved_cursor do
Storage.continue_list(client, saved_cursor)
else
Storage.list_page(client, root, recursive: true, include_deleted: true, limit: 100)
end
with {:ok, page} <- result,
:ok <- commit_page.(page) do
if page.has_more do
scan(client, root, page.cursor, commit_page)
else
{:ok, page.cursor}
end
end
end
endAn application worker loads its checkpoint after acquiring exclusive ownership
of the account and calls the scanner. Here Store is your storage interface,
not a Magpie service:
defmodule MyApp.DropboxScan do
def run(account) do
MyApp.Store.with_account_lease(account, fn ->
client = MyApp.DropboxClients.for_account(account)
root = MyApp.Store.root(account)
saved = MyApp.Store.cursor(account)
MyApp.DropboxRecovery.run_locked(client, account, root, saved)
end)
end
endImplement with_account_lease/2 with a database lock or renewable distributed
lease covering the entire scan and rebuild. A process-local lock is insufficient
across nodes; a lease must fence stale workers and prevent old checkpoints from
overwriting newer ones. apply_page_and_checkpoint/2 must roll back on failure.
Do not persist page.cursor when an entry fails. Retry from the last committed
cursor; repeated entries and duplicate deletion notifications must be harmless.
The runnable document search example
already demonstrates external SQLite cursors and transactional reconciliation
using the unchanged low-level endpoints.
Recovery when Dropbox invalidates the cursor
A 409 with the top-level reset tag returns
%Magpie.CursorError{reason: :reset, rebuild_required: true, error: api_error}.
Dropbox does not tell you whether the cursor expired or was otherwise invalidated.
A path error or malformed request remains Magpie.Error and needs its own
handling. In particular, HTTP 400 on continue_list is not classified as an
invalid cursor: Dropbox may return plain text without a stable error tag, and
400 alone cannot distinguish cursor corruption from another request error.
Inspect diagnostics and the original payload and choose recovery explicitly.
This is a limitation; only a confirmed 409 reset triggers CursorError.
No recovery request is made automatically.
Rebuild a shadow snapshot from the same root/settings. Publish it and its final cursor atomically only after all pages succeed, removing stale local entries absent from the new snapshot. Keep the previous snapshot on failure. Under the same account lease, an application can implement recovery as follows:
defmodule MyApp.DropboxRecovery do
def run_locked(client, account, root, saved) do
case MagpieGuide.Scanner.scan(
client,
root,
saved,
&MyApp.Store.apply_page_and_checkpoint(account, &1)
) do
{:error, %Magpie.CursorError{rebuild_required: true}} ->
shadow = MyApp.Store.begin_shadow(account)
try do
with {:ok, final} <-
MagpieGuide.Scanner.scan(
client,
root,
nil,
&MyApp.Store.apply_shadow_page(shadow, &1)
),
:ok <- MyApp.Store.publish_shadow(account, shadow, final) do
{:ok, final}
end
after
MyApp.Store.discard_shadow(shadow)
end
result ->
result
end
end
endapply_shadow_page/2 writes only the staging snapshot; publish_shadow/3 swaps
snapshot and checkpoint together. discard_shadow/1 cleans staging data without
deleting a published snapshot. These callbacks belong to the consumer.
Rebuilding recovers present state, not intermediate historical events.
Phoenix/Plug webhook endpoint
Enable files.metadata.read and register your public HTTPS endpoint in the
Dropbox App Console. According to the
official webhook contract,
GET verification must echo challenge with Content-Type: text/plain and
X-Content-Type-Options: nosniff. POST notifications carry account IDs; their
HMAC-SHA256 signature uses the app secret and original request body. Dropbox
expects a response within ten seconds. Notifications may concern changes outside
your particular cursor's subfolder.
This complete standalone Plug endpoint also shows the order to use in your
Phoenix endpoint: put Magpie.Webhook.Plug above Plug.Parsers, so it owns
body reading and halts the matching request before any JSON parser. Phoenix
can use json_decoder: Phoenix.json_library() instead of Jason:
defmodule MyApp.DropboxEndpoint do
use Plug.Builder
plug(Magpie.Webhook.Plug,
path: "/webhooks/dropbox",
app_secret: &MyApp.DropboxJobs.app_secret/0,
notify: &MyApp.DropboxJobs.enqueue/1,
max_body_bytes: 1_048_576
)
plug(Plug.Parsers,
parsers: [:urlencoded, :multipart, :json],
pass: ["*/*"],
json_decoder: Jason
)
endUse remote function captures for callbacks in compiled Plug pipelines; anonymous functions cannot be escaped during compile-time initialization. Read secrets at request time through the remote function, rather than embedding them in the compiled endpoint.
The Plug reads original bytes in chunks and preserves them in
conn.private[:magpie_webhook_raw_body]. Do not place a body parser before it
or recreate the body with Jason.encode!. Plug remains a test-only dependency
of Magpie; Phoenix supplies it, or add {:plug, "~> 1.15"} to a standalone app.
There is no Phoenix dependency. The adapter returns 403 for missing/invalid or
duplicate signatures, 400 for malformed signed payloads/read failures, 413 for
oversized bodies, 503 for enqueue errors, and 200 after notify returns :ok.
Signed JSON objects without list_folder are acknowledged with 200 without
calling notify; the pure API returns {:ok, :ignored} and the Plug records
:ignored in private[:magpie_webhook_notification]. Observe these cases in the
application to discover unsupported formats. This is not Business/team support.
Empty account lists also receive 200 without invoking the callback. Recognized
but malformed list_folder notifications still receive 400.
Resolved secrets must be non-empty binaries. Invalid secret-function results and
unexpected notify returns raise explicit ArgumentError without including
returned values. This makes configuration/programming errors visible to the host
instead of disguising them as invalid signatures. Callback exceptions propagate
to the hosting application. Other paths pass through; unsupported methods on the webhook path return 405.
For another framework use Magpie.Webhook.challenge/1 to obtain the response
map and Magpie.Webhook.notification(raw_body, signature, app_secret) to obtain
{:ok, accounts}, {:ok, :ignored} or
{:error, :invalid_signature | :invalid_payload}.
valid_signature?/3 is available separately. Extra JSON fields are ignored;
list_folder.accounts must be a list of non-empty strings (empty is valid).
Order and duplicates are preserved. There is no replay protection in the
signature, nor a per-file event ID to deduplicate.
Background processing, optionally with Oban
Your callback must durably enqueue all notified accounts quickly and return
:ok, or return {:error, reason} so the endpoint can signal failure. Do not
scan synchronously or return success for an unpersisted task. Repeated requests
can enqueue the same account; serialize account scans and make page writes
idempotent. Keep a periodic reconciliation job for missed notifications.
For an application that already has Oban installed, migrated and supervised,
configure a dropbox queue and use this recipe. These application modules are
integration templates; Magpie does not install Oban or a database:
defmodule MyApp.DropboxJobs do
def app_secret, do: System.fetch_env!("DROPBOX_APP_SECRET")
def enqueue(accounts) do
# MyApp.Repo must be the same repo configured for Oban.
case MyApp.Repo.transaction(fn ->
Enum.each(accounts, fn account ->
case %{account: account} |> MyApp.DropboxWorker.new() |> Oban.insert() do
{:ok, _job} -> :ok
{:error, reason} -> MyApp.Repo.rollback(reason)
end
end)
end) do
{:ok, :ok} -> :ok
{:error, reason} -> {:error, reason}
end
end
end
defmodule MyApp.DropboxWorker do
use Oban.Worker, queue: :dropbox, max_attempts: 10
@impl Oban.Worker
def perform(%Oban.Job{args: %{"account" => account}}) do
case MyApp.DropboxScan.run(account) do
{:ok, _cursor} ->
:ok
{:error, %Magpie.Error{} = error} ->
if Magpie.Error.not_found?(error), do: {:cancel, error}, else: {:error, error}
{:error, reason} ->
{:error, reason}
end
end
endThe transaction rolls back all inserts if any enqueue fails. Use the same Ecto
repo as Oban; inserts outside that transaction would not have this guarantee.
Repeated successful deliveries can still enqueue duplicates, so serialize scans
and apply pages idempotently. DropboxScan.run/1 holds the account lease through
both scanning and recovery. Confirmed not_found errors cancel the job: the
application must resolve a removed root or configuration before rescheduling.
Other failures retain Oban retries. Tune permanent-error handling for your app.
Optional Oban uniqueness can reduce duplicate
jobs but does not replace an account lock. Be careful coalescing notifications
while a worker is running: schedule another scan or keep a durable dirty marker
so changes near job completion are not lost. Consult
Oban installation for the consumer's setup.
Offline consumer tests
Use per-client Req.Test configuration so no real credentials are involved. This complete test is extracted and run by the library's suite:
defmodule MyApp.DropboxConsumerTest do
use ExUnit.Case, async: true
test "polls from an externally saved cursor" do
client =
Magpie.Client.new("fake-token", req_options: [plug: {Req.Test, __MODULE__}], retry: false)
Req.Test.stub(__MODULE__, fn conn ->
assert conn.request_path == "/2/files/list_folder/continue"
assert Jason.decode!(Req.Test.raw_body(conn)) == %{"cursor" => "previous"}
Req.Test.json(conn, %{"entries" => [], "cursor" => "saved", "has_more" => false})
end)
assert {:ok, page} = Magpie.Storage.continue_list(client, "previous")
assert page.cursor == "saved"
end
endBuild signed Plug.Test.conn requests with deliberately formatted raw JSON and
assert private.magpie_webhook_raw_body equals those exact bytes. Stub the
enqueue callback, assert it receives every account, and ensure failed signatures
never invoke it. See test/webhook_test.exs and the
testing guide for ownership allowances when jobs run in another
process. Guide tests extract the scanner, recovery, account-leased scan, endpoint
and consumer test, plus the incremental snippet in the README. Fake application
stores exercise the full endpoint → queued account → leased scan → reset recovery
flow. Oban-specific inserts and worker code remain an integration recipe checked
against the official API; the default suite has no Oban or database dependency.