-
Notifications
You must be signed in to change notification settings - Fork 1
feat: URL 추출기 (url_extractor) 도구 추가 #1496
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/new-analysis-tools-6840956748808657165
Are you sure you want to change the base?
Changes from 6 commits
7359d52
750c4b1
0584656
8db683d
5e220ef
687f472
9c5bc0f
e0209af
f4710ca
a83f237
23ef83e
49bbfa4
5135a21
260db8f
dc9ff84
542c1b9
2e25b16
1bdb0c7
6e37926
14e4630
4a465d3
5014fb3
a1ffc7e
46ff31d
d6f8db1
7df1823
a0e877e
977bfca
cc9591d
17b732a
f1c7f6d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -769,6 +769,62 @@ async def uuid_v4_generator_handler(params: Dict[str, Any]) -> Dict[str, str]: | |
| ) | ||
|
|
||
|
|
||
| _URL_PATTERN = re.compile(r"https?://[^\s<>\"']+", re.IGNORECASE) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
In rich-text email containing a smart-quoted link such as Useful? React with 👍 / 👎. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This compile call supplies AGENTS.md reference: AGENTS.md:L687-L687 Useful? React with 👍 / 👎. |
||
| _PROSE_TRAILING_PUNCTUATION = ".,;:!?" | ||
|
|
||
|
|
||
| def _trim_url_candidate(candidate: str, wrapping_openers: str) -> str: | ||
| """Remove only closing delimiters proven by adjacent opening wrappers.""" | ||
| delimiters = (("(", ")"), ("[", "]"), ("{", "}")) | ||
| excess = { | ||
| closer: wrapping_openers.count(opener) | ||
| for opener, closer in delimiters | ||
| } | ||
| without_prose = candidate.rstrip(_PROSE_TRAILING_PUNCTUATION) | ||
| end = len(without_prose) | ||
| while end and excess.get(without_prose[end - 1], 0): | ||
| excess[without_prose[end - 1]] -= 1 | ||
| end -= 1 | ||
| return without_prose[:end] if end < len(without_prose) else candidate | ||
|
devin-ai-integration[bot] marked this conversation as resolved.
|
||
|
|
||
| async def url_extractor_handler(params: Dict[str, Any]) -> Dict[str, list[str]]: | ||
| text = params["text"] | ||
| if len(text) > ANALYSIS_TEXT_MAX_CHARS: | ||
| raise ValueError( | ||
| f"Analysis text must not exceed {ANALYSIS_TEXT_MAX_CHARS} characters" | ||
| ) | ||
| urls: list[str] = [] | ||
| seen: set[str] = set() | ||
| for match in _URL_PATTERN.finditer(text): | ||
| wrapper_start = match.start() | ||
| while wrapper_start and text[wrapper_start - 1] in "([{": | ||
| wrapper_start -= 1 | ||
|
devin-ai-integration[bot] marked this conversation as resolved.
|
||
| candidate = _trim_url_candidate( | ||
| match.group(), text[wrapper_start : match.start()] | ||
| ) | ||
|
seonghobae marked this conversation as resolved.
|
||
| try: | ||
| parsed = urllib.parse.urlsplit(candidate) | ||
| _ = parsed.port # validate a declared port without requiring one | ||
| valid = parsed.hostname is not None | ||
| except ValueError: | ||
| valid = False | ||
| if valid and candidate not in seen: | ||
| seen.add(candidate) | ||
| urls.append(candidate) | ||
| return {"urls": urls} | ||
|
|
||
|
|
||
| registry.register( | ||
| ToolInfo( | ||
| code="url_extractor", | ||
| name="URL 추출기 (URL Extractor)", | ||
| description="텍스트 본문에서 HTTP 및 HTTPS URL을 추출합니다.", | ||
| category="유틸리티", | ||
| parameters={"text": "string"}, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Declaring the required AGENTS.md reference: AGENTS.md:L292-L293 Useful? React with 👍 / 👎. |
||
| ), | ||
| url_extractor_handler, | ||
| ) | ||
|
|
||
|
|
||
| @router.get("/tools", response_model=list[ToolInfo]) | ||
| def get_tools() -> list[ToolInfo]: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When URLs are adjacent without whitespace, this greedy character class consumes the later scheme as part of the first candidate. For example, Markdown such as
[https://a.example](https://b.example)becomes one candidate; the nearbyurlsplitvalidation rejects that candidate, so both URLs are omitted. Stop at Markdown delimiters or restart matching at an embedded HTTP(S) scheme, and cover this case inbackend/tests/test_tools_api.py.Useful? React with 👍 / 👎.