Serve Craft App release info for ArtCraft Launcher (#71)
GET /v1/craft_apps/release_info reports the latest release of every
Craft App (and ArtCraft Launcher itself), so launchers stop polling
GitHub directly. A background thread polls GitHub and keeps a shared
cache.
Polling (threads/poll_craft_app_releases_thread):
- One round at startup (after 0–30 s of jitter, so a fleet deploy is
spread out), then every CRAFT_APPS_RELEASE_POLL_INTERVAL_SECS (30 min
default, 5 min minimum) with ±10% jitter
- One request per app (GET /repos/{repo}/releases/latest) with ETag
revalidation and 1 s between requests
- Rate limits stop the round and wait for GitHub's reset, backing off
2×, 4×, … the interval up to CRAFT_APPS_RELEASE_MAX_BACKOFF_SECS (4 h);
rounds where every request failed retry after 1, 2, 4… min, never more
often, up to the interval
- Per app all-or-nothing: either its release and app_last_fetched are
stored, or nothing about it changes
- CRAFT_APPS_RELEASE_POLLING_ENABLED, CRAFT_APPS_RELEASE_GITHUB_TOKEN
and CRAFT_APPS_RELEASE_APPS (key=Name=org/repo,…) configure it
Cache (state/craft_apps_release_cache): Arc<RwLock<BTreeMap<..>>> of
typed entries. release_info.last_fetched is the oldest app_last_fetched,
absent until every app has been fetched. Apps are listed from the start,
without a release or app_last_fetched until fetched.
The endpoint never fails, sets Cache-Control: public, max-age=60, and
returns empty app_renames, repo_renames and important_notices for now.
API types live in artcraft_api_defs::craft_apps::release_info, with each
asset's OS / arch / package type parsed from its file name. reqwest
moves from a dev-dependency to a dependency of storyteller-web.
Tests: cache freshness, asset-name parsing for every published file
type, untrusted-answer filtering, backoff, and an ignored live round
against GitHub that also writes a sample the launcher's parser is tested
against.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
42b1872760
commit
f37f82b400
22 files changed
+1110
No files matched your search
@@ -84,6 +84,11 @@ report a local update as a deployed update.
|
||||
|
||||
Organize for top-to-bottom reading. Important things first, details later.
|
||||
|
||||
- **`mod.rs` holds module declarations only** (`pub mod foo;`, plus re-exports if needed). Never
|
||||
put functions, structs, enums, constants, impls or tests in a `mod.rs`: give each its own file
|
||||
in that directory (for example `threads/poll_foo_thread/poll_foo.rs`, declared from
|
||||
`threads/poll_foo_thread/mod.rs`).
|
||||
|
||||
- **Constants** at the top (after imports)
|
||||
- **Structs/enums** next; outer structs above inner sub-structs
|
||||
- **API types** in order: Request, Response, Error
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
pub mod release_info;
|
||||
@@ -0,0 +1,203 @@
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde_derive::{Deserialize, Serialize};
|
||||
use url::Url;
|
||||
use utoipa::ToSchema;
|
||||
|
||||
// ── GET /v1/craft_apps/release_info ──
|
||||
//
|
||||
// The latest release of every Craft App (PhotoCraft, VectorCraft, …, and ArtCraft Launcher
|
||||
// itself), as storyteller-web last fetched it from GitHub, plus renames and notices for the
|
||||
// ArtCraft Launcher. The launcher polls this instead of GitHub; GitHub is its fallback.
|
||||
//
|
||||
// Compatibility: clients must ignore fields they don't know, and treat every field as optional.
|
||||
// New fields may be added; existing ones keep their meaning.
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct GetCraftAppsReleaseInfoResponse {
|
||||
pub success: bool,
|
||||
|
||||
pub release_info: CraftAppsReleaseInfo,
|
||||
|
||||
/// Apps that changed their name (for example PdfCraft → DocuCraft).
|
||||
pub app_renames: Vec<CraftAppRename>,
|
||||
|
||||
/// Apps whose GitHub repository moved (for example `storytold/photocraft` → `art/photocraft`).
|
||||
pub repo_renames: Vec<CraftAppRepoRename>,
|
||||
|
||||
pub important_notices: CraftAppsImportantNotices,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppsReleaseInfo {
|
||||
/// Keyed by app key: the app's lowercase machine name, as used in its binaries, asset file
|
||||
/// names and `ai.storyteller.<key>` bundle id (`photocraft`, `cadcraft`, `artcraft-launcher`).
|
||||
/// Not the display name and not the repo name. Clients should ignore keys they don't know.
|
||||
pub apps: BTreeMap<String, CraftAppReleaseInfo>,
|
||||
|
||||
/// When the least recently fetched app was fetched: every app's data is at least this fresh.
|
||||
/// Absent until every app has been fetched at least once.
|
||||
pub last_fetched: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppReleaseInfo {
|
||||
/// Display name (`PhotoCraft`).
|
||||
pub name: String,
|
||||
|
||||
/// GitHub repository, `organization/project` (`storytold/photocraft`).
|
||||
pub repo: String,
|
||||
|
||||
/// The latest published, non-draft, non-prerelease release. Absent when the repository has no
|
||||
/// release yet, or when it hasn't been fetched yet (see `app_last_fetched`).
|
||||
pub latest_release: Option<CraftAppRelease>,
|
||||
|
||||
/// When this app's release data was last fetched successfully. Absent if it never was; then
|
||||
/// `latest_release` says nothing and clients should ask GitHub themselves.
|
||||
pub app_last_fetched: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppRelease {
|
||||
/// Git tag (`v0.5.0`).
|
||||
pub tag: String,
|
||||
|
||||
/// The tag without its leading `v` (`0.5.0`, `0.6.0-rc.1`).
|
||||
pub version: String,
|
||||
|
||||
/// Release title (`PhotoCraft v0.5.0`).
|
||||
pub name: String,
|
||||
|
||||
pub published_at: Option<DateTime<Utc>>,
|
||||
|
||||
/// The release's page on GitHub.
|
||||
pub html_url: String,
|
||||
|
||||
pub prerelease: bool,
|
||||
|
||||
/// Release notes (Markdown), truncated to 64 KiB.
|
||||
pub notes_markdown: String,
|
||||
|
||||
pub assets: Vec<CraftAppReleaseAsset>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppReleaseAsset {
|
||||
/// File name (`photocraft-0.5.0-windows-x64.msi`).
|
||||
pub name: String,
|
||||
|
||||
pub size_bytes: u64,
|
||||
|
||||
/// `https://github.com/<repo>/releases/download/<tag>/<name>`.
|
||||
pub download_url: String,
|
||||
|
||||
/// Lowercase hex SHA-256, when GitHub published one for the file.
|
||||
pub sha256: Option<String>,
|
||||
|
||||
/// What the file name says it is, when it follows the Craft App naming scheme
|
||||
/// (`<app>-<version>-<os>-<arch>[-portable].<ext>`).
|
||||
pub platform: Option<CraftAppAssetPlatform>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppAssetPlatform {
|
||||
pub os: CraftAppOs,
|
||||
|
||||
/// Absent for files that aren't for one CPU (the web build, checksums).
|
||||
pub arch: Option<CraftAppArch>,
|
||||
|
||||
pub package_type: CraftAppPackageType,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum CraftAppOs {
|
||||
Windows,
|
||||
Macos,
|
||||
Linux,
|
||||
Freebsd,
|
||||
/// The browser build.
|
||||
Web,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum CraftAppArch {
|
||||
X64,
|
||||
X86,
|
||||
Arm64,
|
||||
/// macOS universal binary (Apple silicon and Intel).
|
||||
Universal,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum CraftAppPackageType {
|
||||
/// Windows Installer.
|
||||
Msi,
|
||||
/// Windows portable zip.
|
||||
PortableZip,
|
||||
/// macOS disk image.
|
||||
Dmg,
|
||||
/// macOS command-line tools zip.
|
||||
CliZip,
|
||||
AppImage,
|
||||
/// AppImage delta-update file.
|
||||
AppImageZsync,
|
||||
Deb,
|
||||
Rpm,
|
||||
Flatpak,
|
||||
/// Linux or FreeBSD tarball.
|
||||
TarGz,
|
||||
/// The web (WebAssembly) build.
|
||||
WebZip,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppRename {
|
||||
/// User-facing name before (`PdfCraft`).
|
||||
pub old_name: String,
|
||||
/// Lowercase name before (`pdfcraft`).
|
||||
pub old_name_lower: String,
|
||||
/// User-facing name after (`DocuCraft`).
|
||||
pub new_name: String,
|
||||
/// Lowercase name after (`docucraft`).
|
||||
pub new_name_lower: String,
|
||||
/// A sentence or two to show beside the rename.
|
||||
pub description: String,
|
||||
pub renamed_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppRepoRename {
|
||||
/// `organization/project` before.
|
||||
pub old_name: String,
|
||||
/// `organization/project` after.
|
||||
pub new_name: String,
|
||||
pub renamed_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppsImportantNotices {
|
||||
/// News, shown inline; the user can dismiss each.
|
||||
pub announcements: Vec<CraftAppsAnnouncement>,
|
||||
/// Shown as a popup the user acknowledges.
|
||||
pub alerts: Vec<CraftAppsAlert>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppsAnnouncement {
|
||||
pub title: String,
|
||||
pub body: String,
|
||||
pub created_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
|
||||
pub struct CraftAppsAlert {
|
||||
pub title: String,
|
||||
pub body: String,
|
||||
/// A page the popup can offer to open.
|
||||
pub maybe_web_link: Option<Url>,
|
||||
pub created_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
@@ -2,6 +2,7 @@ pub mod analytics;
|
||||
pub mod api_keys;
|
||||
pub mod characters;
|
||||
pub mod common;
|
||||
pub mod craft_apps;
|
||||
pub mod credits;
|
||||
pub mod folders;
|
||||
pub mod generate;
|
||||
|
||||
@@ -162,6 +162,7 @@ use artcraft_api_defs::moderation::user_daily_spends::user_daily_spends_list::*;
|
||||
use artcraft_api_defs::moderation::top_spenders::list::*;
|
||||
use artcraft_api_defs::moderation::user_spend_events::list::*;
|
||||
use artcraft_api_defs::folders::subfolder::*;
|
||||
use artcraft_api_defs::craft_apps::release_info::*;
|
||||
use artcraft_api_defs::tags::add_media_file_tags::*;
|
||||
use artcraft_api_defs::tags::bulk_add_tags::*;
|
||||
use artcraft_api_defs::tags::bulk_list_media_file_tags::*;
|
||||
@@ -734,6 +735,9 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
|
||||
crate::http_server::endpoints::folders::media_files::bulk_move_folder_media_files_handler::bulk_move_folder_media_files_handler,
|
||||
crate::http_server::endpoints::folders::media_files::bulk_remove_folder_media_files_handler::bulk_remove_folder_media_files_handler,
|
||||
|
||||
// Craft Apps
|
||||
crate::http_server::endpoints::craft_apps::get_craft_apps_release_info_handler::get_craft_apps_release_info_handler,
|
||||
|
||||
// Tags
|
||||
crate::http_server::endpoints::tags::list_tags_handler::list_tags_handler,
|
||||
crate::http_server::endpoints::tags::list_media_file_tags_handler::list_media_file_tags_handler,
|
||||
@@ -1606,6 +1610,20 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
|
||||
ListTaggedMediaFilesSuccessResponse,
|
||||
ListTagsQueryParams,
|
||||
ListTagsSuccessResponse,
|
||||
GetCraftAppsReleaseInfoResponse,
|
||||
CraftAppsReleaseInfo,
|
||||
CraftAppReleaseInfo,
|
||||
CraftAppRelease,
|
||||
CraftAppReleaseAsset,
|
||||
CraftAppAssetPlatform,
|
||||
CraftAppOs,
|
||||
CraftAppArch,
|
||||
CraftAppPackageType,
|
||||
CraftAppRename,
|
||||
CraftAppRepoRename,
|
||||
CraftAppsImportantNotices,
|
||||
CraftAppsAnnouncement,
|
||||
CraftAppsAlert,
|
||||
ListUntaggedMediaFilesQueryParams,
|
||||
ListUntaggedMediaFilesSuccessResponse,
|
||||
MediaFileTagsEntry,
|
||||
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use actix_web::http::header::{CacheControl, CacheDirective};
|
||||
use actix_web::{web, HttpResponse};
|
||||
|
||||
use artcraft_api_defs::craft_apps::release_info::{CraftAppsImportantNotices, GetCraftAppsReleaseInfoResponse};
|
||||
|
||||
use crate::state::server_state::ServerState;
|
||||
|
||||
/// Shared caches (Cloudflare) may serve the same answer for this long: the data changes at most
|
||||
/// once per polling round (30 minutes by default).
|
||||
const CACHE_MAX_AGE_SECS: u32 = 60;
|
||||
|
||||
/// The latest release of every Craft App (and ArtCraft Launcher), as storyteller-web last fetched
|
||||
/// it from GitHub, plus app/repo renames and notices for the launcher. Never fails: before the
|
||||
/// first fetch, apps are listed without a release and without `app_last_fetched`.
|
||||
#[utoipa::path(
|
||||
get,
|
||||
tag = "Craft Apps",
|
||||
path = "/v1/craft_apps/release_info",
|
||||
responses(
|
||||
(status = 200, body = GetCraftAppsReleaseInfoResponse),
|
||||
),
|
||||
)]
|
||||
pub async fn get_craft_apps_release_info_handler(
|
||||
server_state: web::Data<Arc<ServerState>>,
|
||||
) -> HttpResponse {
|
||||
let response = GetCraftAppsReleaseInfoResponse {
|
||||
success: true,
|
||||
release_info: server_state.craft_apps_release_cache.snapshot(),
|
||||
// None yet. When there are, they're listed here (newest first).
|
||||
app_renames: Vec::new(),
|
||||
repo_renames: Vec::new(),
|
||||
important_notices: CraftAppsImportantNotices::default(),
|
||||
};
|
||||
HttpResponse::Ok()
|
||||
.insert_header(CacheControl(vec![CacheDirective::Public, CacheDirective::MaxAge(CACHE_MAX_AGE_SECS)]))
|
||||
.json(response)
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
pub mod get_craft_apps_release_info_handler;
|
||||
@@ -5,6 +5,7 @@ pub mod beta_keys;
|
||||
pub mod billing_fakeyou;
|
||||
pub mod characters;
|
||||
pub mod comments;
|
||||
pub mod craft_apps;
|
||||
pub mod credits;
|
||||
pub mod dev;
|
||||
pub mod download_job;
|
||||
|
||||
+2
@@ -63,6 +63,7 @@ use crate::startup::setup_inference_providers::{
|
||||
use crate::startup::setup_pager::build_pager;
|
||||
use crate::startup::setup_static_feature_flags::setup_static_feature_flags;
|
||||
use crate::state::certs::google_sign_in_cert::GoogleSignInCert;
|
||||
use crate::state::craft_apps_release_cache::CraftAppsReleaseCache;
|
||||
use crate::state::memory_cache::model_token_to_info_cache::ModelTokenToInfoCache;
|
||||
use crate::state::server_state::{
|
||||
Dashboards, DataboxDashboards, DurableInMemoryCaches, EnvConfig, EphemeralInMemoryCaches,
|
||||
@@ -702,6 +703,7 @@ pub(crate) fn build_test_server_state(pool: MySqlPool, media_cdn_override_url: S
|
||||
sort_key_crypto: WebSortKeyCrypto::new("test-sort-key"),
|
||||
opaque_cursors: WebOpaqueCursorEncoderV2::new("test-sort-key"),
|
||||
ip_ban_list: IpBanList::new(),
|
||||
craft_apps_release_cache: CraftAppsReleaseCache::new(),
|
||||
cidr_ban_set: BannedCidrSet::new(),
|
||||
troll_bans: TrollBans {
|
||||
user_tokens: TrollUserBanList::new(),
|
||||
|
||||
+2
@@ -3,6 +3,7 @@ use crate::http_server::routes::application_routes::api_keys_routes::add_api_key
|
||||
use crate::http_server::routes::application_routes::billing_fakeyou_routes::add_billing_fakeyou_routes;
|
||||
use crate::http_server::routes::application_routes::character_routes::add_character_routes;
|
||||
use crate::http_server::routes::application_routes::comments_routes::add_comments_routes;
|
||||
use crate::http_server::routes::application_routes::craft_apps_routes::add_craft_apps_routes;
|
||||
use crate::http_server::routes::application_routes::credits_routes::add_credits_routes;
|
||||
use crate::http_server::routes::application_routes::dev_routes::add_dev_routes;
|
||||
use crate::http_server::routes::application_routes::featured_item_routes::add_featured_item_routes;
|
||||
@@ -76,6 +77,7 @@ where
|
||||
app = add_video_info_routes(app); // /v1/video_info/... (video provenance detection)
|
||||
app = add_analytics_routes(app); // /v1/analytics/...
|
||||
app = add_web_referrals_routes(app); // /v1/web_referrals/...
|
||||
app = add_craft_apps_routes(app); // /v1/craft_apps/... (ArtCraft Launcher release info)
|
||||
app = add_dev_routes(app); // /v1/dev/... — dev-only error/response shape probes
|
||||
|
||||
// User and user-adjacent routes
|
||||
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
use actix_http::body::MessageBody;
|
||||
use actix_service::ServiceFactory;
|
||||
use actix_web::dev::{ServiceRequest, ServiceResponse};
|
||||
use actix_web::{web, App, Error, HttpResponse};
|
||||
|
||||
use crate::http_server::endpoints::craft_apps::get_craft_apps_release_info_handler::get_craft_apps_release_info_handler;
|
||||
|
||||
pub fn add_craft_apps_routes<T, B>(app: App<T>) -> App<T>
|
||||
where
|
||||
B: MessageBody,
|
||||
T: ServiceFactory<
|
||||
ServiceRequest,
|
||||
Config = (),
|
||||
Response = ServiceResponse<B>,
|
||||
Error = Error,
|
||||
InitError = (),
|
||||
>,
|
||||
{
|
||||
app.service(
|
||||
web::scope("/v1/craft_apps")
|
||||
// Release info for ArtCraft Launcher (polled instead of GitHub).
|
||||
.service(
|
||||
web::resource("/release_info")
|
||||
.route(web::get().to(get_craft_apps_release_info_handler))
|
||||
.route(web::head().to(|| HttpResponse::Ok())),
|
||||
),
|
||||
)
|
||||
}
|
||||
@@ -3,6 +3,7 @@ mod api_keys_routes;
|
||||
mod billing_fakeyou_routes;
|
||||
mod character_routes;
|
||||
mod comments_routes;
|
||||
mod craft_apps_routes;
|
||||
mod credits_routes;
|
||||
mod dev_routes;
|
||||
mod featured_item_routes;
|
||||
|
||||
@@ -56,6 +56,8 @@ use crate::startup::setup_metrics::build_metrics;
|
||||
use crate::state::server_state::ServerState;
|
||||
use crate::threads::db_health_checker_thread::db_health_checker_thread::db_health_checker_thread;
|
||||
use crate::threads::expire_login_challenges::expire_login_challenges;
|
||||
use crate::threads::poll_craft_app_releases_thread::config::CraftAppsPollConfig;
|
||||
use crate::threads::poll_craft_app_releases_thread::poll_craft_app_releases::poll_craft_app_releases;
|
||||
use crate::threads::poll_ip_banlist_thread::poll_ip_bans;
|
||||
use crate::threads::poll_model_token_info_thread::poll_model_token_info_thread;
|
||||
|
||||
@@ -150,6 +152,15 @@ async fn main() -> AnyhowResult<()> {
|
||||
poll_model_token_info_thread(model_token_info_cache_clone, mysql_pool_clone).await;
|
||||
});
|
||||
|
||||
info!("Spawning craft app release polling thread.");
|
||||
|
||||
let craft_apps_release_cache_clone = server_state.craft_apps_release_cache.clone();
|
||||
let craft_apps_poll_config = CraftAppsPollConfig::from_env();
|
||||
|
||||
tokio_runtime.spawn(async move {
|
||||
poll_craft_app_releases(craft_apps_release_cache_clone, craft_apps_poll_config).await;
|
||||
});
|
||||
|
||||
// ==================== Metrics worker ==================== //
|
||||
|
||||
let (metrics_collector, maybe_metrics_worker) =
|
||||
|
||||
@@ -41,6 +41,7 @@ use crate::startup::setup_static_feature_flags::setup_static_feature_flags;
|
||||
use crate::startup::setup_stripe_artcraft::setup_stripe_artcraft;
|
||||
use crate::startup::setup_stripe_fakeyou::setup_stripe_fakeyou;
|
||||
use crate::state::certs::google_sign_in_cert::GoogleSignInCert;
|
||||
use crate::state::craft_apps_release_cache::CraftAppsReleaseCache;
|
||||
use crate::state::memory_cache::model_token_to_info_cache::ModelTokenToInfoCache;
|
||||
use crate::state::server_state::{
|
||||
DurableInMemoryCaches, EnvConfig, EphemeralInMemoryCaches,
|
||||
@@ -259,6 +260,7 @@ pub async fn setup_dependencies(server_hostname: &str) -> AnyhowResult<SetupResu
|
||||
}
|
||||
},
|
||||
ip_ban_list,
|
||||
craft_apps_release_cache: CraftAppsReleaseCache::new(),
|
||||
cidr_ban_set,
|
||||
troll_bans: TrollBans {
|
||||
user_tokens: user_token_troll_bans,
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
//! The latest release of every Craft App, as last fetched from GitHub by
|
||||
//! `poll_craft_app_releases_thread`, served by `GET /v1/craft_apps/release_info`.
|
||||
//!
|
||||
//! ## App keys
|
||||
//!
|
||||
//! Apps are keyed by their **app key**: the app's lowercase machine name, the same string the
|
||||
//! Craft Apps use for their binaries, asset file names and `ai.storyteller.<key>` bundle ids.
|
||||
//! For example `photocraft`, `pdfcraft`, `cadcraft`, and `artcraft-launcher` for ArtCraft Launcher
|
||||
//! itself. The key is not the display name (`PhotoCraft`) and not the GitHub repo
|
||||
//! (`storytold/photocraft`, whose project name can differ: the launcher's repo is
|
||||
//! `storytold/craft-launcher`).
|
||||
//!
|
||||
//! The keys come from the polling config (`poll_craft_app_releases_thread::config`: the built-in
|
||||
//! list, or `CRAFT_APPS_RELEASE_APPS`) and are served unchanged as the keys of
|
||||
//! `release_info.apps`, which ArtCraft Launcher matches against its own app ids.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::{Arc, PoisonError, RwLock};
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
|
||||
use artcraft_api_defs::craft_apps::release_info::{CraftAppRelease, CraftAppReleaseInfo, CraftAppsReleaseInfo};
|
||||
|
||||
/// Shared between the polling thread and the HTTP handlers. Cheap to clone.
|
||||
#[derive(Clone, Default)]
|
||||
pub struct CraftAppsReleaseCache {
|
||||
/// App key (lowercase machine name, e.g. `photocraft`; see the module docs) → what's known
|
||||
/// about that app. A `BTreeMap` so the endpoint lists apps in a stable order.
|
||||
inner: Arc<RwLock<BTreeMap<String, CachedCraftApp>>>,
|
||||
}
|
||||
|
||||
/// One app: what's known about it, and the ETag to revalidate it with.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct CachedCraftApp {
|
||||
/// Display name (`PhotoCraft`).
|
||||
pub name: String,
|
||||
|
||||
/// GitHub repository, `organization/project` (`storytold/photocraft`).
|
||||
pub repo: String,
|
||||
|
||||
/// The latest published release. `None` both before the first fetch and when the repository
|
||||
/// has no release; `app_last_fetched` tells the two apart.
|
||||
pub latest_release: Option<CraftAppRelease>,
|
||||
|
||||
/// When this app's release was last fetched (or confirmed unchanged) successfully. `None`
|
||||
/// until the first success.
|
||||
pub app_last_fetched: Option<DateTime<Utc>>,
|
||||
|
||||
/// GitHub's ETag for the last successful answer, sent back as `If-None-Match` so an unchanged
|
||||
/// release costs a `304`.
|
||||
pub etag: Option<String>,
|
||||
}
|
||||
|
||||
impl CraftAppsReleaseCache {
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
/// Makes sure every app in `apps` has an entry, so the endpoint reports apps that haven't been
|
||||
/// fetched yet. Each tuple is `(app key, display name, repo)`, for example
|
||||
/// `("photocraft", "PhotoCraft", "storytold/photocraft")`. Apps already present keep what
|
||||
/// was fetched for them.
|
||||
pub fn register_apps(&self, apps: &[(String, String, String)]) {
|
||||
let mut map = self.inner.write().unwrap_or_else(PoisonError::into_inner);
|
||||
for (key, name, repo) in apps {
|
||||
map.entry(key.clone()).or_insert_with(|| CachedCraftApp {
|
||||
name: name.clone(),
|
||||
repo: repo.clone(),
|
||||
latest_release: None,
|
||||
app_last_fetched: None,
|
||||
etag: None,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// The ETag of the last successful answer for the app with app key `key` (`photocraft`), to
|
||||
/// send as `If-None-Match`.
|
||||
pub fn etag(&self, key: &str) -> Option<String> {
|
||||
let map = self.inner.read().unwrap_or_else(PoisonError::into_inner);
|
||||
map.get(key).and_then(|app| app.etag.clone())
|
||||
}
|
||||
|
||||
/// Records a complete, successful fetch of the app with app key `key` (`photocraft`): its
|
||||
/// latest release (`None` if it has none). Ignored for keys that were never registered.
|
||||
pub fn store(&self, key: &str, latest_release: Option<CraftAppRelease>, etag: Option<String>, fetched_at: DateTime<Utc>) {
|
||||
let mut map = self.inner.write().unwrap_or_else(PoisonError::into_inner);
|
||||
if let Some(app) = map.get_mut(key) {
|
||||
app.latest_release = latest_release;
|
||||
app.etag = etag;
|
||||
app.app_last_fetched = Some(fetched_at);
|
||||
}
|
||||
}
|
||||
|
||||
/// Records that GitHub confirmed the app with app key `key` (`photocraft`) is unchanged
|
||||
/// (`304 Not Modified`). Ignored for keys that were never registered.
|
||||
pub fn confirm_unchanged(&self, key: &str, fetched_at: DateTime<Utc>) {
|
||||
let mut map = self.inner.write().unwrap_or_else(PoisonError::into_inner);
|
||||
if let Some(app) = map.get_mut(key) {
|
||||
app.app_last_fetched = Some(fetched_at);
|
||||
}
|
||||
}
|
||||
|
||||
/// Everything, as the endpoint serves it: `apps` keyed by app key.
|
||||
pub fn snapshot(&self) -> CraftAppsReleaseInfo {
|
||||
let map = self.inner.read().unwrap_or_else(PoisonError::into_inner);
|
||||
let apps: BTreeMap<String, CraftAppReleaseInfo> = map
|
||||
.iter()
|
||||
.map(|(key, app)| {
|
||||
(key.clone(), CraftAppReleaseInfo {
|
||||
name: app.name.clone(),
|
||||
repo: app.repo.clone(),
|
||||
latest_release: app.latest_release.clone(),
|
||||
app_last_fetched: app.app_last_fetched,
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let last_fetched = least_recently_fetched(&apps);
|
||||
CraftAppsReleaseInfo { apps, last_fetched }
|
||||
}
|
||||
}
|
||||
|
||||
/// The oldest `app_last_fetched`, or `None` while any app has never been fetched.
|
||||
fn least_recently_fetched(apps: &BTreeMap<String, CraftAppReleaseInfo>) -> Option<DateTime<Utc>> {
|
||||
if apps.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let mut oldest: Option<DateTime<Utc>> = None;
|
||||
for app in apps.values() {
|
||||
let fetched = app.app_last_fetched?;
|
||||
oldest = Some(oldest.map_or(fetched, |o| o.min(fetched)));
|
||||
}
|
||||
oldest
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use chrono::TimeZone;
|
||||
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn last_fetched_is_the_oldest_app_and_absent_until_all_are_fetched() {
|
||||
let cache = CraftAppsReleaseCache::new();
|
||||
cache.register_apps(&[
|
||||
("photocraft".into(), "PhotoCraft".into(), "storytold/photocraft".into()),
|
||||
("filmcraft".into(), "FilmCraft".into(), "storytold/filmcraft".into()),
|
||||
]);
|
||||
assert_eq!(cache.snapshot().apps.len(), 2);
|
||||
assert_eq!(cache.snapshot().last_fetched, None);
|
||||
|
||||
let early = Utc.with_ymd_and_hms(2026, 10, 10, 1, 0, 0).unwrap();
|
||||
let late = Utc.with_ymd_and_hms(2026, 10, 10, 2, 0, 0).unwrap();
|
||||
cache.store("photocraft", None, Some("\"abc\"".into()), late);
|
||||
assert_eq!(cache.snapshot().last_fetched, None, "filmcraft hasn't been fetched yet");
|
||||
|
||||
cache.confirm_unchanged("filmcraft", early);
|
||||
assert_eq!(cache.snapshot().last_fetched, Some(early));
|
||||
assert_eq!(cache.etag("photocraft").as_deref(), Some("\"abc\""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn registering_again_keeps_what_was_fetched() {
|
||||
let cache = CraftAppsReleaseCache::new();
|
||||
let apps = [("photocraft".to_string(), "PhotoCraft".to_string(), "storytold/photocraft".to_string())];
|
||||
cache.register_apps(&apps);
|
||||
let now = Utc::now();
|
||||
cache.confirm_unchanged("photocraft", now);
|
||||
cache.register_apps(&apps);
|
||||
assert_eq!(cache.snapshot().apps["photocraft"].app_last_fetched, Some(now));
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
pub mod cached_queries;
|
||||
pub mod flags;
|
||||
pub mod certs;
|
||||
pub mod craft_apps_release_cache;
|
||||
pub mod memory_cache;
|
||||
pub mod server_state;
|
||||
@@ -1,6 +1,7 @@
|
||||
use std::collections::HashSet;
|
||||
|
||||
use crate::configs::app_startup::username_set::UsernameSet;
|
||||
use crate::state::craft_apps_release_cache::CraftAppsReleaseCache;
|
||||
use crate::configs::static_api_tokens::StaticApiTokenSet;
|
||||
use crate::http_server::deprecated_endpoints::categories::tts::list_fully_computed_assigned_tts_categories::list_fully_computed_assigned_tts_categories::ModelTokensByCategoryToken;
|
||||
use crate::http_server::endpoints::media_files::list::list_featured_media_files_handler::ListFeaturedMediaFilesQueryParams;
|
||||
@@ -124,6 +125,9 @@ pub struct ServerState {
|
||||
|
||||
pub caches: InMemoryCaches,
|
||||
|
||||
/// The latest Craft App releases, kept fresh by `poll_craft_app_releases_thread`.
|
||||
pub craft_apps_release_cache: CraftAppsReleaseCache,
|
||||
|
||||
pub google_sign_in_cert: GoogleSignInCert,
|
||||
|
||||
pub temp_dir_creator: ScopedTempDirCreator,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
pub mod db_health_checker_thread;
|
||||
pub mod expire_login_challenges;
|
||||
pub mod poll_craft_app_releases_thread;
|
||||
pub mod poll_ip_banlist_thread;
|
||||
pub mod poll_model_token_info_thread;
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
//! How storyteller-web polls GitHub for Craft App releases, from the environment.
|
||||
//!
|
||||
//! | Variable | Default | Meaning |
|
||||
//! |-------------------------------------------|----------------|-------------------------------------------------|
|
||||
//! | `CRAFT_APPS_RELEASE_POLLING_ENABLED` | `true` | Poll at all (the endpoint still answers) |
|
||||
//! | `CRAFT_APPS_RELEASE_POLL_INTERVAL_SECS` | `1800` (30 m) | Time between rounds; at least 300 |
|
||||
//! | `CRAFT_APPS_RELEASE_MAX_BACKOFF_SECS` | `14400` (4 h) | Longest wait after repeated failures/rate limits |
|
||||
//! | `CRAFT_APPS_RELEASE_GITHUB_TOKEN` | (none) | Raises GitHub's limit from 60 to 5000 per hour |
|
||||
//! | `CRAFT_APPS_RELEASE_APPS` | built-in list | `key=Name=org/repo,…` to replace the app list |
|
||||
//!
|
||||
//! Every instance in the fleet polls on its own, so the defaults are deliberately slow: one round
|
||||
//! is one request per app, and without a token GitHub allows 60 an hour per IP address.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
const MIN_INTERVAL_SECS: u64 = 300;
|
||||
|
||||
/// The Craft Apps the launcher knows: `(app key, display name, GitHub repo)`. The app key is
|
||||
/// the app's lowercase machine name; it keys `CraftAppsReleaseCache` and the endpoint's
|
||||
/// `release_info.apps`, and must match the app id ArtCraft Launcher uses. ArtCraft Launcher is
|
||||
/// included (key `artcraft-launcher`, repo `storytold/craft-launcher`) so it can check for its
|
||||
/// own updates.
|
||||
const DEFAULT_APPS: &[(&str, &str, &str)] = &[
|
||||
("photocraft", "PhotoCraft", "storytold/photocraft"),
|
||||
("vectorcraft", "VectorCraft", "storytold/vectorcraft"),
|
||||
("filmcraft", "FilmCraft", "storytold/filmcraft"),
|
||||
("lightcraft", "LightCraft", "storytold/lightcraft"),
|
||||
("pdfcraft", "PdfCraft", "storytold/pdfcraft"),
|
||||
("effectcraft", "EffectCraft", "storytold/effectcraft"),
|
||||
("designcraft", "DesignCraft", "storytold/designcraft"),
|
||||
("wordcraft", "WordCraft", "storytold/wordcraft"),
|
||||
("deckcraft", "DeckCraft", "storytold/deckcraft"),
|
||||
("gridcraft", "GridCraft", "storytold/gridcraft"),
|
||||
("cadcraft", "CADCraft", "storytold/cadcraft"),
|
||||
("soundcraft", "SoundCraft", "storytold/soundcraft"),
|
||||
("artcraft-launcher", "ArtCraft Launcher", "storytold/craft-launcher"),
|
||||
];
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct CraftAppsPollConfig {
|
||||
pub enabled: bool,
|
||||
pub interval: Duration,
|
||||
pub max_backoff: Duration,
|
||||
pub github_token: Option<String>,
|
||||
/// `(app key, display name, repo)`; see [`DEFAULT_APPS`].
|
||||
pub apps: Vec<(String, String, String)>,
|
||||
}
|
||||
|
||||
impl CraftAppsPollConfig {
|
||||
pub fn from_env() -> Self {
|
||||
let interval_secs = easyenv::get_env_num::<u64>("CRAFT_APPS_RELEASE_POLL_INTERVAL_SECS", 1800).unwrap_or(1800).max(MIN_INTERVAL_SECS);
|
||||
let max_backoff_secs = easyenv::get_env_num::<u64>("CRAFT_APPS_RELEASE_MAX_BACKOFF_SECS", 14_400).unwrap_or(14_400).max(interval_secs);
|
||||
let github_token = easyenv::get_env_string_optional("CRAFT_APPS_RELEASE_GITHUB_TOKEN")
|
||||
.map(|t| t.trim().to_string())
|
||||
.filter(|t| !t.is_empty());
|
||||
let apps = easyenv::get_env_string_optional("CRAFT_APPS_RELEASE_APPS")
|
||||
.map(|v| parse_app_list(&v))
|
||||
.filter(|apps| !apps.is_empty())
|
||||
.unwrap_or_else(default_apps);
|
||||
Self {
|
||||
enabled: easyenv::get_env_bool_or_default("CRAFT_APPS_RELEASE_POLLING_ENABLED", true),
|
||||
interval: Duration::from_secs(interval_secs),
|
||||
max_backoff: Duration::from_secs(max_backoff_secs),
|
||||
github_token,
|
||||
apps,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn default_apps() -> Vec<(String, String, String)> {
|
||||
DEFAULT_APPS.iter().map(|(k, n, r)| (k.to_string(), n.to_string(), r.to_string())).collect()
|
||||
}
|
||||
|
||||
/// Reads `key=Name=org/repo,…` (app key, display name, repo). Keys are lowercased. Malformed
|
||||
/// entries are skipped.
|
||||
fn parse_app_list(value: &str) -> Vec<(String, String, String)> {
|
||||
value
|
||||
.split(',')
|
||||
.filter_map(|entry| {
|
||||
let mut parts = entry.trim().splitn(3, '=');
|
||||
let key = parts.next()?.trim().to_lowercase();
|
||||
let name = parts.next()?.trim().to_string();
|
||||
let repo = parts.next()?.trim().to_string();
|
||||
let valid_repo = repo.split('/').count() == 2 && !repo.starts_with('/') && !repo.ends_with('/');
|
||||
(!key.is_empty() && !name.is_empty() && valid_repo).then_some((key, name, repo))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn reads_an_app_list_and_skips_bad_entries() {
|
||||
let apps = parse_app_list(" PhotoCraft=PhotoCraft=storytold/photocraft, junk, x=X=noslash, k=K=org/repo ");
|
||||
assert_eq!(apps, vec![
|
||||
("photocraft".to_string(), "PhotoCraft".to_string(), "storytold/photocraft".to_string()),
|
||||
("k".to_string(), "K".to_string(), "org/repo".to_string()),
|
||||
]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_default_list_has_the_suite_and_the_launcher() {
|
||||
let apps = default_apps();
|
||||
assert_eq!(apps.len(), 13);
|
||||
assert!(apps.iter().any(|(k, _, r)| k == "artcraft-launcher" && r == "storytold/craft-launcher"));
|
||||
}
|
||||
}
|
||||
+277
@@ -0,0 +1,277 @@
|
||||
//! One request to GitHub for an app's latest release, and reading the answer.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use wreq::header::{ACCEPT, AUTHORIZATION, ETAG, IF_NONE_MATCH, RETRY_AFTER, USER_AGENT};
|
||||
use wreq::StatusCode;
|
||||
use serde_derive::Deserialize;
|
||||
|
||||
use artcraft_api_defs::craft_apps::release_info::{
|
||||
CraftAppArch, CraftAppAssetPlatform, CraftAppOs, CraftAppPackageType, CraftAppRelease, CraftAppReleaseAsset,
|
||||
};
|
||||
|
||||
const API_BASE: &str = "https://api.github.com";
|
||||
const MAX_NOTES_BYTES: usize = 64 * 1024;
|
||||
const MAX_ASSETS: usize = 200;
|
||||
const MAX_BODY_BYTES: usize = 4 * 1024 * 1024;
|
||||
|
||||
/// What one request produced.
|
||||
#[derive(Debug)]
|
||||
pub enum FetchOutcome {
|
||||
/// The latest release (`None`: the repository has no published release).
|
||||
Fetched { release: Option<CraftAppRelease>, etag: Option<String> },
|
||||
/// `If-None-Match` matched: what we have is current.
|
||||
NotModified,
|
||||
/// GitHub refused: out of quota until this time, if it said.
|
||||
RateLimited { retry_at: Option<DateTime<Utc>> },
|
||||
/// Network error, unexpected status or unreadable answer. Nothing is stored.
|
||||
Failed(String),
|
||||
}
|
||||
|
||||
pub fn build_client() -> wreq::Result<wreq::Client> {
|
||||
wreq::Client::builder()
|
||||
.connect_timeout(Duration::from_secs(10))
|
||||
.timeout(Duration::from_secs(30))
|
||||
.build()
|
||||
}
|
||||
|
||||
/// `GET /repos/{repo}/releases/latest`, revalidated with `etag`.
|
||||
pub async fn fetch_latest_release(client: &wreq::Client, repo: &str, etag: Option<&str>, token: Option<&str>) -> FetchOutcome {
|
||||
let url = format!("{API_BASE}/repos/{repo}/releases/latest");
|
||||
let mut request = client
|
||||
.get(&url)
|
||||
.header(USER_AGENT, "storyteller-web (craft apps release poller)")
|
||||
.header(ACCEPT, "application/vnd.github+json")
|
||||
.header("X-GitHub-Api-Version", "2022-11-28");
|
||||
if let Some(etag) = etag {
|
||||
request = request.header(IF_NONE_MATCH, etag);
|
||||
}
|
||||
if let Some(token) = token {
|
||||
request = request.header(AUTHORIZATION, format!("Bearer {token}"));
|
||||
}
|
||||
let response = match request.send().await {
|
||||
Ok(r) => r,
|
||||
Err(err) => return FetchOutcome::Failed(format!("request failed: {err}")),
|
||||
};
|
||||
let status = response.status();
|
||||
let header = |name: &str| response.headers().get(name).and_then(|v| v.to_str().ok()).map(str::to_string);
|
||||
match status {
|
||||
StatusCode::NOT_MODIFIED => FetchOutcome::NotModified,
|
||||
StatusCode::NOT_FOUND => FetchOutcome::Fetched { release: None, etag: None },
|
||||
StatusCode::FORBIDDEN | StatusCode::TOO_MANY_REQUESTS => {
|
||||
let remaining = header("x-ratelimit-remaining");
|
||||
let reset = header("x-ratelimit-reset").and_then(|r| r.parse::<i64>().ok()).and_then(|s| DateTime::from_timestamp(s, 0));
|
||||
let retry_after = header(RETRY_AFTER.as_str())
|
||||
.and_then(|r| r.parse::<i64>().ok())
|
||||
.map(|s| Utc::now() + chrono::Duration::seconds(s.clamp(0, 86_400)));
|
||||
if status == StatusCode::TOO_MANY_REQUESTS || remaining.as_deref() == Some("0") || retry_after.is_some() {
|
||||
FetchOutcome::RateLimited { retry_at: retry_after.or(reset) }
|
||||
} else {
|
||||
FetchOutcome::Failed(format!("HTTP {status}"))
|
||||
}
|
||||
}
|
||||
StatusCode::OK => {
|
||||
let etag = header(ETAG.as_str());
|
||||
let body = match response.bytes().await {
|
||||
Ok(b) if b.len() <= MAX_BODY_BYTES => b,
|
||||
Ok(b) => return FetchOutcome::Failed(format!("answer too large ({} bytes)", b.len())),
|
||||
Err(err) => return FetchOutcome::Failed(format!("unreadable answer: {err}")),
|
||||
};
|
||||
match parse_release(&body, repo) {
|
||||
Ok(release) => FetchOutcome::Fetched { release: Some(release), etag },
|
||||
Err(err) => FetchOutcome::Failed(err),
|
||||
}
|
||||
}
|
||||
other => FetchOutcome::Failed(format!("HTTP {other}")),
|
||||
}
|
||||
}
|
||||
|
||||
/// The parts of GitHub's release object we use. Everything optional: GitHub's answer is input.
|
||||
#[derive(Deserialize)]
|
||||
struct GithubRelease {
|
||||
#[serde(default)]
|
||||
tag_name: Option<String>,
|
||||
#[serde(default)]
|
||||
name: Option<String>,
|
||||
#[serde(default)]
|
||||
html_url: Option<String>,
|
||||
#[serde(default)]
|
||||
published_at: Option<DateTime<Utc>>,
|
||||
#[serde(default)]
|
||||
prerelease: bool,
|
||||
#[serde(default)]
|
||||
draft: bool,
|
||||
#[serde(default)]
|
||||
body: Option<String>,
|
||||
#[serde(default)]
|
||||
assets: Vec<GithubAsset>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct GithubAsset {
|
||||
#[serde(default)]
|
||||
name: Option<String>,
|
||||
#[serde(default)]
|
||||
size: u64,
|
||||
#[serde(default)]
|
||||
browser_download_url: Option<String>,
|
||||
/// `sha256:<hex>` on assets uploaded since mid-2025.
|
||||
#[serde(default)]
|
||||
digest: Option<String>,
|
||||
}
|
||||
|
||||
/// Reads a release, keeping only downloads that really belong to `repo`.
|
||||
pub fn parse_release(body: &[u8], repo: &str) -> Result<CraftAppRelease, String> {
|
||||
let raw: GithubRelease = serde_json::from_slice(body).map_err(|e| format!("unreadable release: {e}"))?;
|
||||
if raw.draft {
|
||||
return Err("the latest release is a draft".into());
|
||||
}
|
||||
let tag = raw.tag_name.filter(|t| !t.is_empty() && t.len() <= 64).ok_or("the release has no tag")?;
|
||||
let releases_page = format!("https://github.com/{repo}/releases/");
|
||||
let html_url = raw.html_url.filter(|u| u.starts_with(&releases_page)).unwrap_or_else(|| format!("{releases_page}tag/{tag}"));
|
||||
let download_prefix = format!("https://github.com/{repo}/releases/download/");
|
||||
let assets = raw
|
||||
.assets
|
||||
.into_iter()
|
||||
.filter_map(|a| {
|
||||
let name = a.name.filter(|n| valid_file_name(n))?;
|
||||
let download_url = a.browser_download_url.filter(|u| u.starts_with(&download_prefix))?;
|
||||
let sha256 = a.digest.as_deref().and_then(|d| d.strip_prefix("sha256:")).filter(|h| h.len() == 64 && h.bytes().all(|b| b.is_ascii_hexdigit())).map(str::to_ascii_lowercase);
|
||||
let platform = parse_platform(&name);
|
||||
Some(CraftAppReleaseAsset { name, size_bytes: a.size, download_url, sha256, platform })
|
||||
})
|
||||
.take(MAX_ASSETS)
|
||||
.collect();
|
||||
Ok(CraftAppRelease {
|
||||
version: tag.strip_prefix(['v', 'V']).unwrap_or(&tag).to_string(),
|
||||
name: raw.name.filter(|n| !n.is_empty()).map(|n| truncate(&n, 200)).unwrap_or_else(|| tag.clone()),
|
||||
tag,
|
||||
published_at: raw.published_at,
|
||||
html_url,
|
||||
prerelease: raw.prerelease,
|
||||
notes_markdown: raw.body.map(|b| truncate(&b, MAX_NOTES_BYTES)).unwrap_or_default(),
|
||||
assets,
|
||||
})
|
||||
}
|
||||
|
||||
/// Reads `<app>-<version>-<os>-<arch>[-portable].<ext>`, `<app>-cli-<version>-macos-universal.zip`
|
||||
/// and `<app>-web-<version>.zip`. `None` for anything else (checksums, unknown files).
|
||||
pub fn parse_platform(name: &str) -> Option<CraftAppAssetPlatform> {
|
||||
if name.contains("-web-") && name.ends_with(".zip") {
|
||||
return Some(CraftAppAssetPlatform { os: CraftAppOs::Web, arch: None, package_type: CraftAppPackageType::WebZip });
|
||||
}
|
||||
const KINDS: [(&str, CraftAppPackageType); 10] = [
|
||||
("-portable.zip", CraftAppPackageType::PortableZip),
|
||||
(".AppImage.zsync", CraftAppPackageType::AppImageZsync),
|
||||
(".AppImage", CraftAppPackageType::AppImage),
|
||||
(".tar.gz", CraftAppPackageType::TarGz),
|
||||
(".msi", CraftAppPackageType::Msi),
|
||||
(".dmg", CraftAppPackageType::Dmg),
|
||||
(".deb", CraftAppPackageType::Deb),
|
||||
(".rpm", CraftAppPackageType::Rpm),
|
||||
(".flatpak", CraftAppPackageType::Flatpak),
|
||||
(".zip", CraftAppPackageType::CliZip),
|
||||
];
|
||||
let (base, package_type) = KINDS.iter().find_map(|(suffix, kind)| name.strip_suffix(suffix).map(|b| (b, *kind)))?;
|
||||
let mut parts = base.rsplitn(3, '-');
|
||||
let (arch, os) = (parts.next()?, parts.next()?);
|
||||
let os = match os {
|
||||
"windows" => CraftAppOs::Windows,
|
||||
"macos" => CraftAppOs::Macos,
|
||||
"linux" => CraftAppOs::Linux,
|
||||
"freebsd" => CraftAppOs::Freebsd,
|
||||
_ => return None,
|
||||
};
|
||||
let arch = match arch {
|
||||
"x64" | "x86_64" => CraftAppArch::X64,
|
||||
"x86" => CraftAppArch::X86,
|
||||
"arm64" | "aarch64" => CraftAppArch::Arm64,
|
||||
"universal" => CraftAppArch::Universal,
|
||||
_ => return None,
|
||||
};
|
||||
// A bare `.zip` is only the macOS command-line tools.
|
||||
if package_type == CraftAppPackageType::CliZip && !(os == CraftAppOs::Macos && base.contains("-cli-")) {
|
||||
return None;
|
||||
}
|
||||
Some(CraftAppAssetPlatform { os, arch: Some(arch), package_type })
|
||||
}
|
||||
|
||||
fn valid_file_name(name: &str) -> bool {
|
||||
!name.is_empty()
|
||||
&& name.len() <= 200
|
||||
&& !name.starts_with('.')
|
||||
&& !name.chars().any(|c| c.is_control() || matches!(c, '/' | '\\' | ':' | '<' | '>' | '"' | '|' | '?' | '*'))
|
||||
}
|
||||
|
||||
fn truncate(s: &str, max: usize) -> String {
|
||||
if s.len() <= max {
|
||||
return s.to_string();
|
||||
}
|
||||
let mut end = max;
|
||||
while !s.is_char_boundary(end) {
|
||||
end -= 1;
|
||||
}
|
||||
format!("{}…", &s[..end])
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const REPO: &str = "storytold/photocraft";
|
||||
const HASH: &str = "8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1";
|
||||
|
||||
fn release_json(download_url: &str) -> String {
|
||||
format!(
|
||||
r#"{{"tag_name":"v0.5.0","name":"PhotoCraft v0.5.0","draft":false,"prerelease":false,
|
||||
"html_url":"https://github.com/storytold/photocraft/releases/tag/v0.5.0",
|
||||
"published_at":"2026-10-08T14:52:50Z","body":"Notes","unknown_field":[1,2,3],
|
||||
"assets":[{{"name":"photocraft-0.5.0-windows-x64.msi","size":7,"browser_download_url":"{download_url}","digest":"sha256:{HASH}"}}]}}"#
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reads_a_release() {
|
||||
let r = parse_release(release_json("https://github.com/storytold/photocraft/releases/download/v0.5.0/photocraft-0.5.0-windows-x64.msi").as_bytes(), REPO).unwrap();
|
||||
assert_eq!((r.tag.as_str(), r.version.as_str(), r.name.as_str()), ("v0.5.0", "0.5.0", "PhotoCraft v0.5.0"));
|
||||
assert_eq!(r.assets.len(), 1);
|
||||
assert_eq!(r.assets[0].sha256.as_deref(), Some(HASH));
|
||||
let p = r.assets[0].platform.as_ref().unwrap();
|
||||
assert_eq!((p.os, p.arch, p.package_type), (CraftAppOs::Windows, Some(CraftAppArch::X64), CraftAppPackageType::Msi));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn drops_downloads_that_point_elsewhere_and_rejects_garbage() {
|
||||
for url in ["https://evil.example/x.msi", "https://github.com/storytold/other/releases/download/v1/x.msi"] {
|
||||
assert!(parse_release(release_json(url).as_bytes(), REPO).unwrap().assets.is_empty(), "{url}");
|
||||
}
|
||||
assert!(parse_release(b"<html>", REPO).is_err());
|
||||
assert!(parse_release(br#"{"assets":[]}"#, REPO).is_err());
|
||||
assert!(parse_release(br#"{"tag_name":"v1.0.0","draft":true}"#, REPO).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reads_every_asset_name_the_suite_publishes() {
|
||||
let cases = [
|
||||
("photocraft-0.5.0-macos-universal.dmg", Some((CraftAppOs::Macos, Some(CraftAppArch::Universal), CraftAppPackageType::Dmg))),
|
||||
("photocraft-cli-0.5.0-macos-universal.zip", Some((CraftAppOs::Macos, Some(CraftAppArch::Universal), CraftAppPackageType::CliZip))),
|
||||
("photocraft-0.5.0-windows-arm64-portable.zip", Some((CraftAppOs::Windows, Some(CraftAppArch::Arm64), CraftAppPackageType::PortableZip))),
|
||||
("photocraft-0.5.0-windows-x86.msi", Some((CraftAppOs::Windows, Some(CraftAppArch::X86), CraftAppPackageType::Msi))),
|
||||
("photocraft-0.5.0-linux-aarch64.AppImage", Some((CraftAppOs::Linux, Some(CraftAppArch::Arm64), CraftAppPackageType::AppImage))),
|
||||
("photocraft-0.5.0-linux-x86_64.AppImage.zsync", Some((CraftAppOs::Linux, Some(CraftAppArch::X64), CraftAppPackageType::AppImageZsync))),
|
||||
("photocraft-0.5.0-linux-x86_64.deb", Some((CraftAppOs::Linux, Some(CraftAppArch::X64), CraftAppPackageType::Deb))),
|
||||
("photocraft-0.5.0-linux-x86_64.rpm", Some((CraftAppOs::Linux, Some(CraftAppArch::X64), CraftAppPackageType::Rpm))),
|
||||
("photocraft-0.5.0-linux-x86_64.flatpak", Some((CraftAppOs::Linux, Some(CraftAppArch::X64), CraftAppPackageType::Flatpak))),
|
||||
("photocraft-0.5.0-freebsd-x86_64.tar.gz", Some((CraftAppOs::Freebsd, Some(CraftAppArch::X64), CraftAppPackageType::TarGz))),
|
||||
("photocraft-0.2.0-rc.1-linux-x86_64.tar.gz", Some((CraftAppOs::Linux, Some(CraftAppArch::X64), CraftAppPackageType::TarGz))),
|
||||
("photocraft-web-0.5.0.zip", Some((CraftAppOs::Web, None, CraftAppPackageType::WebZip))),
|
||||
("SHA256SUMS.txt", None),
|
||||
("photocraft-0.5.0-plan9-x64.msi", None),
|
||||
];
|
||||
for (name, want) in cases {
|
||||
let got = parse_platform(name).map(|p| (p.os, p.arch, p.package_type));
|
||||
assert_eq!(got, want, "{name}");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
pub mod config;
|
||||
pub mod github;
|
||||
pub mod poll_craft_app_releases;
|
||||
+229
@@ -0,0 +1,229 @@
|
||||
//! Polls GitHub for the latest release of every Craft App and keeps it in
|
||||
//! [`CraftAppsReleaseCache`], which `GET /v1/craft_apps/release_info` serves to ArtCraft Launcher
|
||||
//! (so thousands of launchers don't each poll GitHub).
|
||||
//!
|
||||
//! Every instance of storyteller-web polls on its own, so it is gentle: one round at startup,
|
||||
//! then one per `CRAFT_APPS_RELEASE_POLL_INTERVAL_SECS` (30 minutes by default) with jitter, one
|
||||
//! request per app with ETag revalidation, and growing waits after rate limits or failures.
|
||||
//! Each app is all-or-nothing: either its latest release is stored with a fresh
|
||||
//! `app_last_fetched`, or nothing about it changes.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use log::{error, info, warn};
|
||||
|
||||
use crate::state::craft_apps_release_cache::CraftAppsReleaseCache;
|
||||
use crate::threads::poll_craft_app_releases_thread::config::CraftAppsPollConfig;
|
||||
use crate::threads::poll_craft_app_releases_thread::github;
|
||||
use crate::threads::poll_craft_app_releases_thread::github::FetchOutcome;
|
||||
|
||||
/// Spreads a fleet's first round over this window, so a deploy doesn't hit GitHub all at once.
|
||||
const STARTUP_JITTER: Duration = Duration::from_secs(15);
|
||||
/// Pause between the requests of one round.
|
||||
const BETWEEN_REQUESTS: Duration = Duration::from_millis(1_000);
|
||||
/// First wait after a round where every request failed (network trouble); doubles up to the
|
||||
/// normal interval.
|
||||
const FIRST_FAILURE_RETRY: Duration = Duration::from_secs(60);
|
||||
/// Extra margin after GitHub's announced rate-limit reset.
|
||||
const RESET_MARGIN: Duration = Duration::from_secs(60);
|
||||
|
||||
pub async fn poll_craft_app_releases(cache: CraftAppsReleaseCache, config: CraftAppsPollConfig) {
|
||||
cache.register_apps(&config.apps);
|
||||
if !config.enabled {
|
||||
info!("Craft app release polling is disabled (CRAFT_APPS_RELEASE_POLLING_ENABLED=false).");
|
||||
return;
|
||||
}
|
||||
let client = match github::build_client() {
|
||||
Ok(client) => client,
|
||||
Err(err) => {
|
||||
error!("Craft app release polling can't start: no HTTP client: {:?}", err);
|
||||
return;
|
||||
}
|
||||
};
|
||||
info!(
|
||||
"Polling GitHub for {} craft app releases every {:?} (token: {}).",
|
||||
config.apps.len(), config.interval, config.github_token.is_some(),
|
||||
);
|
||||
tokio::time::sleep(jittered(STARTUP_JITTER, 1.0)).await;
|
||||
|
||||
let mut consecutive_trouble = 0u32;
|
||||
loop {
|
||||
let round = poll_round(&client, &cache, &config).await;
|
||||
let wait = next_wait(&config, &round, &mut consecutive_trouble, Utc::now());
|
||||
info!(
|
||||
"Craft app releases: {} fetched, {} unchanged, {} failed{}; next round in {:?}.",
|
||||
round.fetched, round.unchanged, round.failed,
|
||||
if round.rate_limited.is_some() { ", rate limited" } else { "" },
|
||||
wait,
|
||||
);
|
||||
tokio::time::sleep(wait).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// What one round did.
|
||||
#[derive(Debug, Default)]
|
||||
struct RoundResult {
|
||||
fetched: usize,
|
||||
unchanged: usize,
|
||||
failed: usize,
|
||||
/// GitHub refused; `Some(Some(t))` when it said when to come back.
|
||||
rate_limited: Option<Option<DateTime<Utc>>>,
|
||||
}
|
||||
|
||||
async fn poll_round(client: &wreq::Client, cache: &CraftAppsReleaseCache, config: &CraftAppsPollConfig) -> RoundResult {
|
||||
let mut round = RoundResult::default();
|
||||
for (i, (key, _, repo)) in config.apps.iter().enumerate() {
|
||||
if i > 0 {
|
||||
tokio::time::sleep(BETWEEN_REQUESTS).await;
|
||||
}
|
||||
let etag = cache.etag(key);
|
||||
match github::fetch_latest_release(client, repo, etag.as_deref(), config.github_token.as_deref()).await {
|
||||
FetchOutcome::Fetched { release, etag } => {
|
||||
cache.store(key, release, etag, Utc::now());
|
||||
round.fetched += 1;
|
||||
}
|
||||
FetchOutcome::NotModified => {
|
||||
cache.confirm_unchanged(key, Utc::now());
|
||||
round.unchanged += 1;
|
||||
}
|
||||
FetchOutcome::RateLimited { retry_at } => {
|
||||
warn!("GitHub rate limit reached polling {repo}; retry after {:?}.", retry_at);
|
||||
round.rate_limited = Some(retry_at);
|
||||
// The rest of the round would be refused too.
|
||||
break;
|
||||
}
|
||||
FetchOutcome::Failed(why) => {
|
||||
warn!("Couldn't fetch the latest release of {repo}: {why}");
|
||||
round.failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
round
|
||||
}
|
||||
|
||||
/// How long to wait before the next round. Rate limits wait for GitHub's reset and back off
|
||||
/// further each time; rounds where everything failed retry sooner, but never faster than a
|
||||
/// minute, doubling up to the normal interval.
|
||||
fn next_wait(config: &CraftAppsPollConfig, round: &RoundResult, consecutive_trouble: &mut u32, now: DateTime<Utc>) -> Duration {
|
||||
let base = if let Some(retry_at) = round.rate_limited {
|
||||
*consecutive_trouble = consecutive_trouble.saturating_add(1);
|
||||
let until_reset = retry_at
|
||||
.and_then(|t| (t - now).to_std().ok())
|
||||
.map(|d| d + RESET_MARGIN)
|
||||
.unwrap_or(Duration::ZERO);
|
||||
let backoff = scaled(config.interval, *consecutive_trouble);
|
||||
until_reset.max(backoff).min(config.max_backoff)
|
||||
} else if round.fetched == 0 && round.unchanged == 0 && round.failed > 0 {
|
||||
*consecutive_trouble = consecutive_trouble.saturating_add(1);
|
||||
scaled(FIRST_FAILURE_RETRY, *consecutive_trouble).min(config.interval)
|
||||
} else {
|
||||
*consecutive_trouble = 0;
|
||||
config.interval
|
||||
};
|
||||
jittered(base, 0.1)
|
||||
}
|
||||
|
||||
/// `base × 2^(n-1)`, without overflow.
|
||||
fn scaled(base: Duration, n: u32) -> Duration {
|
||||
base.saturating_mul(1u32 << n.saturating_sub(1).min(16))
|
||||
}
|
||||
|
||||
/// `base` ± `fraction`, so a fleet doesn't poll in lockstep. With `fraction == 1.0`, anywhere in
|
||||
/// `0..=2×base`; used for the startup spread.
|
||||
fn jittered(base: Duration, fraction: f64) -> Duration {
|
||||
let factor = 1.0 + fraction * (rand::random::<f64>() * 2.0 - 1.0);
|
||||
base.mul_f64(factor.max(0.0))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::threads::poll_craft_app_releases_thread::config::default_apps;
|
||||
|
||||
use super::*;
|
||||
|
||||
fn config() -> CraftAppsPollConfig {
|
||||
CraftAppsPollConfig {
|
||||
enabled: true,
|
||||
interval: Duration::from_secs(1800),
|
||||
max_backoff: Duration::from_secs(14_400),
|
||||
github_token: None,
|
||||
apps: default_apps(),
|
||||
}
|
||||
}
|
||||
|
||||
fn within(d: Duration, around: Duration) -> bool {
|
||||
d >= around.mul_f64(0.89) && d <= around.mul_f64(1.11)
|
||||
}
|
||||
|
||||
/// Live: one real round against GitHub (two apps), then the endpoint's response, written to
|
||||
/// `CRAFT_APPS_SAMPLE_OUT` when set (ArtCraft Launcher's parser reads it back).
|
||||
/// `cargo test -p storyteller-web --bin storyteller-web live_round -- --ignored`
|
||||
#[tokio::test]
|
||||
#[ignore = "needs network access"]
|
||||
async fn live_round_fetches_real_releases() {
|
||||
use artcraft_api_defs::craft_apps::release_info::{CraftAppsImportantNotices, GetCraftAppsReleaseInfoResponse};
|
||||
let mut config = config();
|
||||
config.apps.retain(|(key, _, _)| key == "photocraft" || key == "artcraft-launcher");
|
||||
let cache = CraftAppsReleaseCache::new();
|
||||
cache.register_apps(&config.apps);
|
||||
let client = github::build_client().unwrap();
|
||||
let round = poll_round(&client, &cache, &config).await;
|
||||
assert_eq!(round.fetched + round.unchanged, 2, "{round:?}");
|
||||
// A second round revalidates with ETags.
|
||||
let again = poll_round(&client, &cache, &config).await;
|
||||
assert_eq!(again.fetched + again.unchanged, 2, "{again:?}");
|
||||
let info = cache.snapshot();
|
||||
assert!(info.last_fetched.is_some());
|
||||
let photocraft = info.apps["photocraft"].latest_release.as_ref().unwrap();
|
||||
assert!(photocraft.assets.iter().any(|a| a.platform.is_some() && a.sha256.is_some()));
|
||||
let response = GetCraftAppsReleaseInfoResponse {
|
||||
success: true,
|
||||
release_info: info,
|
||||
app_renames: Vec::new(),
|
||||
repo_renames: Vec::new(),
|
||||
important_notices: CraftAppsImportantNotices::default(),
|
||||
};
|
||||
let json = serde_json::to_string_pretty(&response).unwrap();
|
||||
if let Ok(path) = std::env::var("CRAFT_APPS_SAMPLE_OUT") {
|
||||
std::fs::write(path, &json).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_good_round_waits_the_interval_and_resets_trouble() {
|
||||
let mut trouble = 3;
|
||||
let round = RoundResult { fetched: 5, unchanged: 8, ..RoundResult::default() };
|
||||
assert!(within(next_wait(&config(), &round, &mut trouble, Utc::now()), Duration::from_secs(1800)));
|
||||
assert_eq!(trouble, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rate_limits_wait_for_the_reset_and_back_off_further_each_time() {
|
||||
let now = Utc::now();
|
||||
let mut trouble = 0;
|
||||
// GitHub says come back in 50 minutes: wait that (plus margin), more than the interval.
|
||||
let round = RoundResult { rate_limited: Some(Some(now + chrono::Duration::minutes(50))), ..RoundResult::default() };
|
||||
assert!(within(next_wait(&config(), &round, &mut trouble, now), Duration::from_secs(51 * 60)));
|
||||
// Again and again, with no reset time: 2×, 4×, … the interval, capped at the maximum.
|
||||
let round = RoundResult { rate_limited: Some(None), ..RoundResult::default() };
|
||||
assert!(within(next_wait(&config(), &round, &mut trouble, now), Duration::from_secs(3600)));
|
||||
assert!(within(next_wait(&config(), &round, &mut trouble, now), Duration::from_secs(7200)));
|
||||
for _ in 0..10 {
|
||||
assert!(next_wait(&config(), &round, &mut trouble, now) <= Duration::from_secs(14_400).mul_f64(1.11));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_round_where_everything_failed_retries_sooner_but_not_in_a_tight_loop() {
|
||||
let now = Utc::now();
|
||||
let mut trouble = 0;
|
||||
let round = RoundResult { failed: 13, ..RoundResult::default() };
|
||||
assert!(within(next_wait(&config(), &round, &mut trouble, now), Duration::from_secs(60)));
|
||||
assert!(within(next_wait(&config(), &round, &mut trouble, now), Duration::from_secs(120)));
|
||||
for _ in 0..20 {
|
||||
let w = next_wait(&config(), &round, &mut trouble, now);
|
||||
assert!(w >= Duration::from_secs(53) && w <= Duration::from_secs(1800).mul_f64(1.11), "{w:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user