SSE & AI Requests Report Improvements

doc/gsoc/2025/manas_hejmadi.md

@@ -39,11 +39,8 @@ All in all, these completion of these improvements will establish apidash as a m
 `Associated Pull Request`: [#857](https://github.com/foss42/apidash/pull/857)
 
 Initially, the entire networking constructs that apidash relied on was fully written inside a module named `apidash_core`. 
-The networking code was fairly advanced including support for GraphQL, request cancellations and a lot of other good features. Howeever, as it was tightly coupled with apidash, we were unable to allow the rest of the flutter developer community to use these features.
+The networking code was fairly advanced including support for GraphQL, request cancellations and a lot of other good features. However, as it was tightly coupled with apidash, we were unable to allow the rest of the flutter developer community to use these features. We believe in giving back to the open source community whenever we can and hence the mentors and I decided to refactor everything into a new package. 
-At apidash, we believe in giving back to the open source community and hence the mentors and I decided to refactor everything into a new package. 
+During discussions, I came up with the name `better_networking` and we envisioned it to  be the go-to package for everything related to networking for a flutter application.
-During discussions, I came up with the name `better_networking`. We envisioned it to  be the go-to package for everything related to networking for a flutter application.
-
-This proved to be a great decision, as we were able to separate it completely, publish it on [pub.dev](https://pub.dev/packages/better_networking), and achieve over 95% code coverage through isolated testing.
 
 This is an example of how better_networking simplifies request handling
 
@@ -69,6 +66,12 @@ final (resp, duration, err) = await sendHttpRequest(
 cancelHttpRequest('unique-request-id');
 ```
 
+This proved to be a great decision, as we were able to separate it completely, publish it on [pub.dev](https://pub.dev/packages/better_networking), and achieve over 95% code coverage through isolated testing.
+
+Code coverage before refactor:
+![Code Coverage Report](./images/codecovold.png)
+
+Code coverage after Refactor:
 ![Code Coverage Report](./images/bnetlcov.png)
 
 ---
@@ -77,10 +80,15 @@ cancelHttpRequest('unique-request-id');
 `Associated Pull Request`: [#861](https://github.com/foss42/apidash/pull/861)
 
 ![Code Coverage Report](./images/sse_ex1.png)
-SSE Support was a long pending [issue](https://github.com/foss42/apidash/issues/116) (since 2024) and hence the mentors asked me to see if i was able to implement SSE support into `better_networking` and simultaneously into `apidash` itself. The implementations suggested by other contributors in the past involved creation of SSE as a completely new request type.
-However, I did not agree with this approach as SSE is not a fundamentally separate request type like GraphQL. Hence, I wrote up a quick demo with SSE implemented within the existing apidash foundation code. The mentors were impressed with this approach as it was far more maintainable and sensible than creating new models for it.
 
-Rewrote the original implementation of `sendHttpRequest` in terms of this new SSE handler
+SSE Support was a long pending [issue](https://github.com/foss42/apidash/issues/116) (since 2024). Once I was done with the `better_networking` package creation, the mentors asked me to look into how i can implement SSE within the package and by extension into apidash. After doing some research and a review into the existing PRs by other contributors for this feature, I noticed that everyone created new Request and Response Models for SSE in code. 
+
+However, I did not agree with this approach as SSE is just a different content-type is not a fundamentally separate request type like GraphQL. 
+To demonstrate this, I wrote up a quick demo with SSE baked into the existing apidash foundations.
+
+This new mechanism is very simple and elegant. Basically, every request in apidash is executed in streaming mode using `StreamedResponse` in dart. If the response headers specify a content-type marked as streaming, the listener remains active and statefully saves all incoming values into the sseOutput attribute of the response model. If the content-type does not match any supported streaming type, the listener terminates and the output is returned immediately. In this way, the existing request/response model can handle both Streaming and Normal HTTP Requests
+
+This is an example of how I rewrote the original implementation of `sendHttpRequest` in terms of this new SSE handler
 ```dart
 Future<(HttpResponse?, Duration?, String?)> sendHttpRequest(
   String requestId,
@@ -100,6 +108,7 @@ Future<(HttpResponse?, Duration?, String?)> sendHttpRequest(
   return (output?.$2, output?.$3, output?.$4);
 }
 ```
+The mentors were impressed with this approach as it was far more maintainable and sensible than creating new models specifically for SSE.
 This way, everything stays unified and we reduce the amount of duplication
  
 ---
@@ -108,19 +117,63 @@ This way, everything stays unified and we reduce the amount of duplication
 
 `Associated Pull Request`: [#870](https://github.com/foss42/apidash/pull/870)
 
-With the rapid rise of Generative AI, it became clear that API Dash required a dedicated AI request interface with support for agentic systems. Based on this need, my mentors tasked me with developing a comprehensive AI Requests feature, along with an integrated agent building framework for future agentic integrations within the apidash application
+With the rapid rise of Generative AI, it became clear that API Dash required a dedicated AI request interface with support for agentic systems. Based on this need, my mentors tasked me with developing a comprehensive AI Requests feature
+
+The user initiates a new request by selecting “AI”, then chooses a model and provides the required credentials through the Authorization tab. The request can be further configured by specifying system and user prompts and adjusting parameters such as `temperature`, `topP`, and `streaming or non-streaming mode`. Upon submission, the response is generated and presented either as cleaned plaintext/Markdown or in raw format, based on the user’s selection.
+
 
 ![AI Requests](./images/aireq1.png)
 ![AI Requests](./images/aireq2.png)
 
-The new AI Requests feature supports key capabilities such as remote model import and selection, multi-provider integration, along with support for streaming responses.
+My initial implementation used tightly coupled LLM providers (e.g., gemini, openai) with specific models (e.g., gemini-2.0-flash) through hardcoded enums. These enums were directly referenced in code, which on closer review proved unsustainable. Given the rapid pace of innovation in LLMs, models become obsolete quickly, and maintaining hardcoded enums would require frequent code changes and was looking quite impractical.
+Furthermore, using hardcoded enums prevents runtime dynamic loading, restricting users to only the models we explicitly provide. This limits flexibility and creates a poor experience, especially for advanced users who may need access to less common or custom models.
 
-![LLM Provider Selector](./images/modelselector2.png)
+To address this, we adopted a remote model fetch system, where model identifiers are stored in a `models.json` file within the public apidash repository. Clients fetch this file at runtime, enabling over-the-air updates to model availability. In addition, we added support for custom model identifiers directly within the ModelSelector, giving users full flexibility to configure their own models.
 
-The newly created genai package enables users to build their own agents with features like prompt templating and more, making it simple and efficient to create powerful in-app agents.
+Currently, we support several standard providers—such as Google Gemini, OpenAI, Anthropic, and Ollama—which offers a strong baseline of options while still allowing advanced customization.
+
+![LLM Provider Selector](./images/modelselector1.png)
+
+The AI Requests feature is built on top of the foundational genai package, which serves as the core layer for all AI-related functionality within apidash.
+This package provides the complete set of API callers, methods, and formatters required to abstract away the complexities of interacting with AI tool APIs. By exposing a generalized interface across multiple providers, it eliminates the need to handle provider-specific details directly.
+As a result, developers can easily build features that leverage generative AI without worrying about low-level implementation details—leaving the intricacies of API communication and formatting to the genai package.
+
+Example of simplified usage (model-agnostic, works with any LLM out of the box)
+```dart
+final LLMModel model = LLMProvider.gemini.getLLMByIdentifier('gemini-2.0-flash');
+final ModelController controller = model.provider.modelController;
+
+final payload = controller.inputPayload
+  ..systemPrompt = 'Say YES or NO'
+  ..userPrompt = 'The sun sets in the west'
+  ..credential = 'AIza....';
+
+final genAIRequest = controller.createRequest(model, payload);
+final answer = await GenerativeAI.executeGenAIRequest(model, genAIRequest);
+
+print(answer);
+```
+
+#### Agentic Infrastructure
 
 ![Agentic Infrastructure](./images/llmarch.png)
 
+When developing AI-powered features in any application, the process typically involves steps such as system prompting, data validation, and output formatting. However, repeating this workflow for multiple features while taking care of errors and retry logic quickly becomes very cumbersom. To simplify this, we designed a well-defined architecture for building AI agents directly within code.
+
+The core idea is straightforward: an AI agent in apidash is simply a Dart file containing a class that extends the base class `APIDashAIAgent`, defined as:
+```dart
+abstract class APIDashAIAgent {
+  String get agentName;
+  String getSystemPrompt();
+  Future<bool> validator(String aiResponse);
+  Future<dynamic> outputFormatter(String validatedResponse);
+}
+```
+This base class provides the necessary hooks for implementing an agent. Developers can either rely on the default implementations or override these handlers with custom logic. The result is a fully abstracted, self-contained agent that can be invoked seamlessly from within the application.
+
+These agents operate within an orchestrator and governor framework that manages everything behind the scenes. This design ensures that developers only need to invoke the agent, while background processes handle concerns such as automatic retries, exponential backoff, and error recovery seamlessly. This saves a lot of time and effort and allows developers to spend more time on improving their actual feature implementation.
+
+
 #### Sample Agent Code
 
 ```dart
@@ -132,11 +185,13 @@ class SimpleFuncGenerator extends APIDashAIAgent {
 
   @override
   String getSystemPrompt() {
-    return """You are a function generator.
+    return """you are a programming language function generator. 
-Given API details (REQDATA) and a programming language (TARGET_LANGUAGE),
+    your only task is to take whatever requirement is provided and convert
-create a method named `func` that performs the API call.
+    it into a valid function named func in the provided programming language
-Return only the code.
+
-""";
+    LANGUAGE: :LANG:
+    REQUIREMENT: :REQUIREMENT:
+"""; //suports templating via :<VARIABLE>:
   }
 
   @override
@@ -155,7 +210,18 @@ Return only the code.
   }
 }
 
+//Calling an agent
+ final res = await APIDashAgentCaller.instance.call(
+    SimpleFuncGenerator(),
+    ref: ref,
+    input: AgentInputs(variables: {
+      'REQUIREMENT': 'take the median of the given array',
+      'TARGET_LANGUAGE': 'python3',
+    }),
+  );
+
 ```
+
 ---
 
 ### Created the API Tool Generator
@@ -204,6 +270,7 @@ This makes use of the Server Driven UI Concept powered by [Stac](https://stac.de
 
 talk about new learnign and overcoming issues
 
+- SSE long text splitting into 2 packets causing errors and how i handled it
 - SDUI Dilemma (cannot use hosted rendering server, cannot use Reflection, cannot bundle flutter sdk), hence had to rely on Server Driven UI using Stac
 - Stac has no support for returning errors
 - Larger UIs tend to generate too long Stac code which gets clipped and leads to faulty output