# mlflow-feedback

**Repository Path**: mirrors_serialx/mlflow-feedback

## Basic Information

- **Project Name**: mlflow-feedback
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-09-20
- **Last Updated**: 2026-07-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# MLflow GenAI Infrastructure Feedback

This repository contains technical feedback and proposed improvements for the MLflow GenAI infrastructure, with accompanying code examples demonstrating the identified issues.

## Issue 1: Evaluation Performance Bottleneck (mlflow_eval_time.py)

### Problem Statement
The current implementation of `mlflow.genai.evaluate()` with `predict_fn` functions introduces unnecessary performance overhead during evaluation workflows. The system samples and tests the first input to verify that `predict_fn` produces a valid `mlflow.trace`, which creates significant delays when evaluating long-running agents.

### Technical Details
The validation process logs the following message:
```
mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset.
```

### Performance Impact
For agent evaluation scenarios where individual `predict_fn()` calls require substantial processing time (e.g., 10 seconds), this validation step effectively doubles the total evaluation duration due to the mandatory initial sample test.

### Cost Impact
The mandatory sample validation incurs unnecessary LLM API costs. Each validation run consumes tokens and API calls that provide no value to the actual evaluation process, resulting in wasted computational resources and increased operational expenses for users running large-scale evaluations.

### Proposed Solutions
1. **Trace Annotation Detection**: Implement a lightweight check to determine if `predict_fn` is properly annotated with `mlflow.trace()` without executing the full prediction pipeline
2. **Skip Flag Option**: Provide a configuration flag allowing users to bypass this validation step entirely when working with known, properly instrumented functions

## Issue 2: Async Support and Resource Management (mlflow_openai_agents_async_eval.py)

### Problem Statement
The `mlflow.genai.evaluate()` API currently lacks native support for asynchronous `predict_fn` functions, creating integration challenges for projects built on async-native frameworks such as OpenAI Agents.

### Technical Challenges

#### 1. Async Function Support
Modern agent implementations, particularly those using OpenAI Agents SDK, are designed with async-first architecture. The current synchronous-only API requires workarounds that complicate codebase integration and reduce development efficiency.

#### 2. Resource Leak Issues
The OpenAI Agents SDK utilizes LiteLLM for third-party model integration. However, LiteLLM's internal connection pool exhibits file descriptor leaks when used within thread pool environments during agent evaluation. This issue has been documented in the upstream repository:

**Reference**: [LiteLLM Issue #13220](https://github.com/BerriAI/litellm/issues/13220)

### Current Workarounds
We have implemented internal solutions to manually close leaked file descriptors, but this approach is suboptimal and introduces maintenance overhead.

### Proposed Enhancements
1. **Native Async Support**: Extend `mlflow.genai.evaluate()` to accept and properly handle async `predict_fn` functions
2. **Resource Management**: Improve connection pool management to prevent file descriptor leaks in multi-threaded evaluation scenarios

## Code Examples
The accompanying Python files demonstrate both issues with reproducible examples that illustrate the performance and resource management challenges described above.