optimization for extracting email and URLs

[armijnhemel]

In case the emails and URLs are extracted it is possible to avoid running the (expensive) regular expression and replace it with a much simpler and cheaper check to filter out large amounts of files (over 50% in my experience).

By first checking for '@' and '://' you can avoid having to run some of these checks, if these characters are not present.

[pombredanne]

@SaravananOffl could this be something you can dig into? This should be not too hard IMHO

[pombredanne]

By first checking for '@' and '://' you can avoid having to run some of these checks, if these characters are not present.

You could include www in the mix as plain scheme-less URLs are also recognized

[SaravananOffl]

@pombredanne you want me to modify some existing code to check for " www ,@, :// " if they're found we can skip some other tests .Am I right?

[pombredanne]

correct: as a prefilter for URL detection (checking for ://, http: or www and as filter for email detection checking for @ such that lines or files without these are not further "regexed" to detect these clues.
You can get familiar with the code in https://github.com/nexB/scancode-toolkit/blob/develop/src/cluecode/finder.py
Other cleanups are welcomed too there, but try to keep it on point.

[SaravananOffl]

@pombredanne I looked into that file you've mentioned above.There are many functions in it .Could you explicitly tell which one i'm supposed to work on.

[pombredanne]

The top level entry points are in https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/api.py#L104 and you can follow the flow from there.
Both email and url end up eventually there https://github.com/nexB/scancode-toolkit/blob/develop/src/cluecode/finder.py#L48

So you may filter lines there.... with the caveats that patterns identified for URLs are quite far ranging including also IP v4/v6 addresses

[SaravananOffl]

@pombredanne I've corrected the regrex expressions for email and ip v4/v6 addresses in finder.py file .What more should I add/modify?

[yash-nisar]

@pombredanne , I'm taking this one as this doesn't have a PR. :)