Purpose:

Notes: missing_from_llm_response

Purpose:

Notes: missing_from_llm_response

Purpose:

Notes: missing_from_llm_response

Purpose: Entry point for the repository summarizer, providing the main function to run the application.

Exports: main

Imports: src.cli:main

Dependencies: Uses the 'src.cli' module to import the 'main' function.

Key Logic: Defines the main entry point for the application, calling the CLI main function.

Notes: Includes core functionality for the application.

Purpose: "This file defines the command-line interface (CLI) for the repository summarizer, including parsing arguments and orchestrating the full pipeline of operations.

Exports: parse_args, build_config_from_args, progress_callback, analyze_commit_history, scan_directory_tree, compute_language_stats, detect_config_files, detect_entry_points

Imports: argparse, tempfile, os, sys, shutil, datetime, huggingface_hub, utils.stdio

Purpose: Contains fundamental data models, configuration, and path management for the repository summarizer.

Exports: OutputPaths, FileInfo, DirectoryInfo, CommitStats, RepoInfo

Imports: from .paths import *, from .models import *, from .config import *

Dependencies: os, datetime, enum, typing, dataclasses

Key Logic: Imports and exports fundamental data models and configuration.

Notes: Includes paths, file and directory information, commit statistics, and repository information.

Purpose: Contains data models representing the fundamental entities used throughout the codebase.

Imports: dataclasses, os, pathlib, typing, src.core.config, src.core.paths, src.core.utils

Important Classes:

• FileInfo
• DirectoryInfo
• CommitStats
• RepoInfo

Dependencies: Uses 'dataclasses', 'os', 'pathlib', 'typing' for data structures and utilities.

Key Logic: Imports necessary modules and dataclasses for core functionality.

Notes: Includes core data models for repository summarization.

Purpose: Centralizes path configuration for repository summarizer, ensuring consistent output locations.

Imports: os, pathlib, typing, dataclasses, src.core.config, src.core.utils

Dependencies: Uses 'os', 'pathlib', 'typing', 'dataclasses' for data structures and utilities.

Key Logic: Imports necessary modules and dataclasses for core functionality.

Notes: Includes centralized output paths for all generated artifacts.

Purpose: Contains the LLM module for repository summarizer, including model client, discovery, selection, and optimization.

Exports: LlamaClient, get_model, reset_model, DiscoveredModel, ModelRole, discover_models, get_default_models, format_model_size

Imports: from .model_client import *, from .model_discovery import *, from .model_selector import *, from .param_optimizer import *

Important Functions:

• interactive_model_selection
• select_models_with_defaults
• get_optimal_params
• check_gpu_available
• normalize_model_name

Dependencies: os, datetime, enum, typing, dataclasses

Key Logic: Imports and exports components for interacting with LLM models.

Notes: Includes components for managing LLM models, including selection and optimization.

Purpose: This file contains functions and classes for discovering and managing local GGUF files, categorizing them by role and size, and printing discovered models.

Exports: DiscoveredModel, ModelRole, discover_models, is_default_analysis, is_default_summarization, print_discovered_models, format_model_size, _extract_quantization

Imports: os, glob, re, dataclasses, enum, pathlib, typing, defaultdict

Key Components:

• DiscoveredModel (class): A class representing information about a discovered local model, including its name, path, size, and inferred role.
• ModelRole (class): An enumeration class for categorizing model roles based on size and capability.
• discover_models (function): A function that scans the ./models/ directory to find available AI models and returns a list of DiscoveredModel objects.
• is_default_analysis (function): A function that checks if a model is a suitable default analysis model based on its name and filename.
• is_default_summarization (function): A function that checks if a model is the default summarization model based on its name and filename.
• print_discovered_models (function): A function that prints a list of discovered models.
• format_model_size (function): A function that formats the size of a model into a human-readable string.
• _extract_quantization (function): A private function that extracts quantization type from a filename.

Data Structures: DiscoveredModel, ModelRole

Algorithms: Role categorization based on model name and filename, size formatting

Configuration: Configuration for the models directory is handled by the DEFAULT_MODELS_DIR constant.

Notes: The file includes private functions that are not intended for public use, such as _extract_quantization and _extract_base_model.

Purpose: Automatically detects hardware capabilities and model characteristics to provide optimal parameters for inference performance.

Exports: get_optimal_params

Imports: os, re, multiprocessing, typing, Literal

Dependencies: The module depends on the 'os', 're', 'multiprocessing', and 'typing' modules for utility functions and data structures.

Key Logic: The module provides a function to get optimal parameters based on model size and available hardware, including CPU and GPU detection.

Notes: The module includes functions for detecting hardware capabilities and model characteristics, and provides optimal parameters for Llama model initialization based on these characteristics.

Purpose: Contains the rendering and output generation module for the repository summarizer.

Exports: generate_directory_tree_mermaid, generate_interactive_diagram, ArtifactPaths, ArtifactManager, render_html_report

Imports: from .visualizer import *, from .artifacts import *, from .html_report import *

Dependencies: os, datetime, enum, typing, dataclasses

Key Logic: Imports and exports components for generating visual diagrams and HTML reports.

Notes: Includes components for generating visual diagrams and HTML reports.

Purpose: This file defines classes and functions for managing and saving artifacts related to repository summarization, including saving and loading various structured outputs.

Exports: ArtifactPaths, ArtifactManager, _ensure_directories, _serialize_datetime, _save_json, _load_json, save_summary_md, _compute_file_stats

Imports: json, os, shutil, datetime, dataclasses, Path, RepoInfo, CommitStats

Key Components:

• ArtifactPaths (class): Centralizes paths for all output artifacts, including directory structure and file names.
• ArtifactManager (class): Manages saving and loading of various structured outputs, ensuring proper directory structure and file naming conventions.

Data Structures: ArtifactPaths, RepoInfo, CommitStats

Algorithms: None explicitly defined in this file

Configuration: Configuration is handled through static paths defined in the module and class attributes. No external configuration files are used.

Notes: The file focuses on the implementation of methods for saving and loading structured outputs, ensuring they are saved in the correct directories and files. It also includes methods for checking the existence of specific artifacts and retrieving them.

Purpose: This file contains functions and classes for rendering an HTML report based on structured artifacts and a reusable HTML template. It handles data transformation and rendering logic, separating HTML structure and styling from Python code.

Exports: _escape_html, _markdown_to_html, close_list, close_table, close_blockquote, process_inline, _load_template, _get_fallback_template

Imports: html, json, os, re, ast, datetime, typing, Path

Key Components:

• _escape_html (function): Safely escapes HTML special characters.
• _markdown_to_html (function): Converts markdown to HTML, handling various markdown elements.
• HTMLReportRenderer (class): Class for rendering the HTML report, encapsulating various rendering methods.

Data Structures: None explicitly defined, but data transformation and rendering are handled.

Algorithms: None explicitly defined, but data transformation logic is implied.

Configuration: No explicit configuration is handled in this file.

Notes: This file focuses on rendering an HTML report by transforming structured data and applying a reusable HTML template. It separates HTML structure and styling from Python code, making the HTML layout maintainable and independent of the data transformation logic.

Purpose: Contains the summarization module for the repository summarizer, including a map-then-reduce pipeline, LLM prompts, batched file processing, and documentation generation.

Exports: BatchSummarizer, SummaryResult, FileForSummary, parse_json_response, run_map_reduce_pipeline, build_triage_prompt, build_batch_summary_prompt, build_file_summary_prompt

Imports: from .batching import *, from .prompts import *, from .doc_builder import *

Dependencies: os, datetime, enum, typing, dataclasses

Key Logic: Imports and exports components for summarizing files and repositories.

Notes: Includes components for summarizing files and repositories, including a map-then-reduce pipeline and documentation generation.

Purpose: This file implements a batched summarization mechanism for repository files, optimizing for efficient LLM calls and caching.

Exports: FileForSummary, SummaryResult, BatchSummarizer, parse_json_response, _read_file_content

Imports: json, os, re, time, logging, dataclasses, pathlib, typing

Key Components:

• FileForSummary (class): A class representing a file to be summarized, containing its path, absolute path, content, size in lines, language, and whether it's large.
• SummaryResult (class): A class representing the result of summarizing files, containing the summaries, errors, cache hits, LLM calls, and token counts.
• BatchSummarizer (class): A class responsible for batch summarization, including methods for logging and summarization.

Data Structures: FileForSummary, SummaryResult

Algorithms: MAP-then-REDUCE pattern, token estimation, file chunking, context overlap, cache lookups, JSON parsing with retry logic

Configuration: SummarizerConfig, DEFAULT_CONFIG

Notes: The file implements a mechanism for batch summarization of repository files, optimizing for efficient LLM calls and caching. It includes a class for summarizing files and a class for batch summarization, each with methods for logging and summarization.

Purpose: This file contains functions and constants for generating optimized prompts for code repository summarization, adhering to specific design principles and rules.

Exports: _as_list, build_json_fix_prompt, estimate_tokens, truncate_content, extract_signatures, build_readme_analysis_prompt, build_global_summary_prompt, build_architecture_prompt

Imports: json, re

Key Components:

• SYSTEM_PROMPT (constant): A detailed system prompt for the code analyzer, enforcing specific rules for JSON output and preventing code generation.
• SYSTEM_PROMPT_DETAILED (constant): A more detailed system prompt for the code analyzer, providing additional rules and requirements.
• _as_list (function): Converts input to a list, handling various data types.
• build_json_fix_prompt (function): Generates a JSON fix prompt for invalid JSON, ensuring it conforms to the specified rules.
• estimate_tokens (function): Estimates the number of tokens in a given text.
• truncate_content (function): Truncates content to a specified maximum number of tokens.
• extract_signatures (function): Extracts function signatures from code content.
• build_readme_analysis_prompt (function): Generates a prompt for analyzing a README file.

Data Structures: List, Dict, str, int

Algorithms: None explicitly defined in the code

Configuration: None explicitly defined in the code

Notes: The functions are designed to generate prompts for summarizing code repositories, with specific rules for JSON output and code generation prevention.

Purpose: Provides heuristic-based and LLM-based file selection for repository analysis, supporting two-stage triage processes.

Exports: quick_triage, llm_triage, scan_repository, interactive_triage, HeuristicFlags, FileMetadata, RepositorySchema, TriageResult

Imports: os, flask, pandas, HeuristicFlags, FileMetadata, RepositorySchema, TriageResult, DirectoryCategory

Dependencies: The module depends on the 'os', 'flask', and 'pandas' packages for utility functions, and on the 'src.triage.schema' and 'src.triage.triage' modules for data models and triage functions.

Key Logic: The module defines functions for heuristic and LLM-based file selection, with the heuristic process using file patterns and reference counts, and the LLM process refining the selection using AI judgment.

Notes: The module contains data models and functions for repository schema, file metadata, and triage processes. It also includes functions for scanning repositories and filtering schemas based on categories and languages.

Purpose: This file contains functions and classes for scanning and building file metadata schema in a repository, including language detection, file categorization, and heuristic flag computation.

Exports: _matches_any_pattern, _should_ignore, _detect_language, _categorize_directory, _compute_heuristic_flags, _count_lines, FileMetadata, RepositorySchema

Imports: os, re, datetime, pathlib, typing, collections, fnmatch, schema

Key Components:

• _matches_any_pattern (function): Determines if a file name matches any of the provided patterns.
• _should_ignore (function): Checks if a file path should be ignored based on the provided ignore patterns.
• _detect_language (function): Detects the programming language of a file based on its extension.
• _categorize_directory (function): Categorizes a directory based on its path relative to the repository root.
• _compute_heuristic_flags (function): Computes heuristic flags for a file based on its metadata.
• _count_lines (function): Counts the number of lines in a file.
• FileMetadata (class): Represents metadata for a file, including its path, language, and other properties.
• RepositorySchema (class): Represents the schema for the repository, including file metadata and other properties.

Data Structures: FileMetadata, RepositorySchema, HeuristicFlags, DirectoryCategory, TriageResult

Algorithms: Language detection via extension mapping, heuristic flag computation based on file metadata, directory categorization based on path relative to the repository root

Configuration: Configurable ignore patterns for file paths, reference counting via simple text search for import detection, language detection via extension mapping (extensible)

Notes: The file contains a mapping of file extensions to programming languages, which is used for language detection. The functions handle various aspects of file metadata, including categorization, heuristic flag computation, and line counting. The imports and exports are organized to ensure that the necessary dependencies are available for the operations.

Purpose: Defines data models for repository file metadata, including categories, flags, and file metadata.

Exports: DirectoryCategory, HeuristicFlags, FileMetadata

Imports: dataclass, field, asdict, datetime, enum, typing, Any

Dependencies: os, datetime, enum, typing, dataclasses

Key Logic: Defines data classes for file metadata and heuristic flags, including methods to convert to dictionaries.

Notes: Includes class definitions for categorizing directories and flags derived from filenames and location.

Purpose: This file contains functions for file triage and heuristic scoring within a repository analysis system.

Exports: _compute_heuristic_score, quick_triage

Imports: DirectoryCategory, FileMetadata, RepositorySchema, TriageResult, scan_repository, os, flask

Key Components:

• _compute_heuristic_score (function): Computes an importance score for a file based on heuristics such as entrypoints, config files, dependency manifests, reference counts, file types, and file size.
• quick_triage (function): Performs a quick triage of the repository, selecting files based on heuristic scoring and returning a triage result.

Data Structures: FileMetadata, DirectoryCategory, RepositorySchema, TriageResult

Algorithms: Heuristic-based scoring algorithm for file importance

Configuration: Score thresholds for heuristics (CRITICAL_THRESHOLD, HIGH_THRESHOLD, MEDIUM_THRESHOLD)

Notes: The quick_triage function uses the heuristic scores to select a maximum of 20 files for triage. The heuristics include checking for entrypoints, config files, dependency manifests, reference counts, file types, and file size.

Purpose: Contains helper functions for various utilities such as file hashing and caching, language/extension detection, CLI formatting and output, and Git operations.

Exports: extension_map, special_names, get_language_for_filename, is_binary_file, should_count_for_language_stats, binary_extensions, excluded_from_stats, set_centered

Imports: os, shutil, re, multiprocessing, platform, torch, subprocess, summary_cache

Dependencies: The module depends on the 'os', 'shutil', 're', 'multiprocessing', 'platform', 'torch', 'subprocess', 'summary_cache', 'cache_key', 'cached_summary', 'get_cache', 'reset_cache', and 'analyze_commit_history' modules for utility functions and data structures.

Key Logic: The module provides functions for various utilities, including file and language detection, CLI formatting, and Git operations.

Notes: The module includes functions for hashing and caching files, detecting file extensions and languages, and formatting CLI output. It also provides functions for analyzing commit history and managing Git operations.

Purpose: This file defines a persistent cache for file summaries using SQLite, keyed by file content hash, prompt version, model identifier, and chunking settings. It aims to avoid re-summarizing unchanged files across runs.

Exports: CacheKey, CachedSummary, normalize_repo_id, repo_cache_dir_name, safe_model_dirname, SummaryCache, get, put

Imports: hashlib, re, json, os, sqlite3, dataclasses, datetime, pathlib

Key Components:

• CacheKey (class): A composite key for cache lookups, consisting of file hash, prompt version, model identifier, and chunk settings.
• CachedSummary (class): A cached file summary entry, containing file path, summary, creation timestamp, and hit count.
• normalize_repo_id (function): Normalizes a repository ID by stripping, converting to lowercase, removing trailing slashes, and removing the '.git' suffix.
• repo_cache_dir_name (function): Generates a directory name for the repository cache based on the normalized repository ID.
• safe_model_dirname (function): Generates a safe directory name for the model based on the model ID.
• SummaryCache (class): Manages the cache, including initialization, schema creation, file hash computation, and summary retrieval and storage.

Data Structures: CacheKey, CachedSummary

Algorithms: None explicitly defined, but includes file hash computation and summary retrieval logic.

Configuration: Handles configuration through CacheConfig and config files, including default values.

Notes: The cache uses SQLite for storage, and the cache directory is generated based on the repository ID and model ID. The cache avoids re-summarizing unchanged files by checking file hashes.

Purpose: Maps file extensions to languages and provides utility functions for determining file types.

Imports: os

Important Functions:

• get_language_for_filename
• is_binary_file
• should_count_for_language_stats

Dependencies: os

Key Logic: Provides a dictionary mapping file extensions to their corresponding languages and utility functions for determining file types.

Notes: Includes utility functions for determining file types and counting language statistics.

Purpose: Provides CLI formatting utilities for repository summarizer, including colored output, frames, and banners.

Exports: set_centered, _terminal_width, _center_line, _banner, _separator, _bold, _info, _starting

Imports: shutil, re, typing, os

Dependencies: The module depends on the 'shutil', 're', and 'typing' modules for utility functions and data structures.

Key Logic: The module offers functions for setting centering behavior, determining terminal width, centering lines, and creating banners, separators, and frames.

Notes: The module includes functions for setting centering behavior, determining terminal width, centering lines, and creating various UI elements for CLI output.