Pluggable framework for syncing third-party content into Readwise Reader. Daemon-on-Docker, secrets via Doppler.
Sources fetch candidate items from external APIs. The Syncer asks Readwise what's already saved (cached in memory),
then writes only the new items via Readwise's /save/ endpoint. APScheduler runs each source on its own
interval. Doppler provides every secret at process start; nothing is committed.
Each sync warms an in-memory URL set from /list/?category=<cat>, scoped to the source's category (video for YouTube, article for github_stars). We never re-save an existing URL, so triaging in Reader is preserved.
3.5s minimum between /save/ calls keeps us under Readwise's 20/min. On 429s we read Retry-After and sleep that long (tenacity-style backoff capped too low for Readwise's ~47s windows).
Every registered source is probe-built at startup. Missing creds → source_skipped warning + skip from the schedule. Adding a source to registry.py never breaks an existing deployment.
Conventional unprefixed env names for SDKs (READWISE_TOKEN, GITHUB_TOKEN, YOUTUBE_OAUTH_*). Internal config uses SYNCRW_. The container's doppler run -- wraps CMD only when DOPPLER_TOKEN is set.
Click any row for responsibilities and what it imports.
class Source(ABC): name: str default_location: str = "later" default_tags: tuple[str, ...] = () readwise_category: str | None = None # scopes the dedup cache @abstractmethod def fetch_candidates(self) -> Iterable[Item]: ...
Each interval, APScheduler fires _run_source(name) for the source whose timer expired. Two sources run on independent timers; they share one ReadwiseClient (and thus one in-memory dedup cache).
/list/?category=<cat>, populates _known_urls. Subsequent ticks in the same daemon process: no-op (idempotent per category).Item(url, title, …)./save/. Tags: source defaults ∪ per-source override from YAML. URL added to _known_urls on success.sync_completed. Errors during a single item are logged with item_create_failed and the loop continues.src/sync_to_readwise/sources/<name>.py implementing Source:
class MySource(Source):
name = "my_source"
default_location = "later"
default_tags = ("my_source",)
readwise_category = "article" # or "video" / etc.
def __init__(self, *, token: str) -> None:
if not token:
raise ValueError("MY_TOKEN must be set.")
self._token = token
def fetch_candidates(self) -> Iterable[Item]:
with httpx.Client(...) as c:
for ... in ...:
yield Item(url=..., source_name="my_source", title=...)
core/config.py Settings:
my_token: SecretStr = Field(default=SecretStr(""), validation_alias="MY_TOKEN")
registry.py:
def _build_my_source(cfg: AppConfig, src_cfg: SourceConfig) -> Source:
return MySource(token=cfg.settings.my_token.get_secret_value())
REGISTRY = {..., "my_source": _build_my_source}
docker-compose.yml and document it in the README's secrets table.source_skipped.No syncer / scheduler / dedup changes needed — the framework picks it up automatically.
chowda, Dockerallenhutchison/sync-to-readwise:latest (Docker Hub).github/workflows/publish.yml on every push to main + vX.Y.Z tags.github/workflows/ci.yml — ruff format + lint + Docker smoke (CLI --help dispatch)docker-entrypoint.sh wraps CMD with doppler run -- when DOPPLER_TOKEN is set; otherwise plain exec/data — holds youtube_token.json (OAuth refresh) and optional config.yaml| Secret | Required | Used by |
|---|---|---|
READWISE_TOKEN | yes | core |
YOUTUBE_OAUTH_CLIENT_ID | youtube only | youtube |
YOUTUBE_OAUTH_CLIENT_SECRET | youtube only | youtube |
GITHUB_TOKEN | github_stars only | github_stars |
# Local dev sync (one-shot, useful for testing a source) doppler run -- docker compose run --rm sync-to-readwise sync-to-readwise sync-once youtube # Run the daemon locally doppler run -- docker compose up -d # YouTube OAuth dance (opens a printed URL; redirect to localhost:8080) doppler run -- docker compose run --rm --service-ports sync-to-readwise sync-to-readwise setup youtube # Tail logs on chowda docker logs --tail 200 -f sync-to-readwise