Backstory

This story begins with me opening vscode with a python file and seeing this autocomplete:

I was in the middle of building a simple text editor,and I thought how could vscode,the most popular code editor in the world, be so unoptimized for programmers?

The autocomplete results were filled with functions I used at most once in my life, with some, like CLD_CONTINUED nowhere to be found, even on the entire public github.

Switching to ty LSP (fastest,open source LSP) from pylance just highlights the problem (basedpyright didnt solve it either).

.

The problem with alphabet sort for code

In many cases,alphabet order is good. For UI, it looks good.. right?

• Google Material - “Lists should be sorted in logical ways that make content easy to scan, such as alphabetical,..,or by user preference”

• uxcel - “Can be alphabetical, numerical, or custom”

• Apple HIG - “Place items to convey their relative importance”

• NN/g - “..,prioritization by importance or frequency are usually better than A-Z listings..,People Rarely Think A–Z”

In programming, you type letter by letter, when you write sys.a you are (almost) certainly not thinking about sys.abiflags or sys.activate_stack_trampoline and if you were thinking about them, you are probably one of the only 10 people in the world who know about them :)

Solutions

Some IDEs, like PyCharm, offer an AI-based autocomplete, this is usually very good autocomplete. However, it’s slow, and usually off by default. When you type code you don’t want your editor to run a full AI model (or even slower, a full LLM) for every letter you type.

Instead of training an AI to detect the rules of autocomplete so it could work with any module, all I actually needed was a simple table of most used functions/attrs for most common modules, which is going to account for most of the code anyway.

So, I used an existing python dataset, called ManyTypes4PyDataset,v1.7 which contains 5.2K Python repositories, and I built a simple python script to count all calls to builtin functions or attributes that happened more than once. That results in a table like this one

..~100,000 rows

However, just dumping this table would result in a big and slow file for no reason, so I designed a binary to make it faster. The Focus: lookup speed.

Hash Score Table format

The first thing that takes up space here is the prefix string. Since the format is designed to be query only, no need to include the actual string (instead,it includes just the hash). The format uses a fast hash, FNV-1a , shifted to be 24-bit.

- Header (magic `HSCT`+version) - 8 bytes
- capacity and slot count - 8 bytes
- Slots hash table 4 bytes repeated:
    - Hash key (FNV-1a >> 8) - 3 bytes
    - frequency - 1 byte

(Using powers of 2 to avoid % operation on lookup)

The frequency is normalized into 1-255 range, so it can be stored with 1 byte.

By this point, I abandoned my python script, and migrated it to go (partly using an LLM, they are trained on go), as python was good as a prototype, but it was too slow to iterate every time on the 5k json files.

Thresholds

Following the steps above gives a ~4.4Mb file, mostly full of things like myClass.A(),that are too specific. creating a classical name filter (A.A) only changes the file size to a ~4.2Mb file.

I added two new thresholds.

1. Filter by total number of calls (or accesses). For example, if os.stat was called 1459 times, it’s probably because it’s popular (raw)
2. Filter by number of projects calling. For example, if a function is called 1500 times only by a single project, it is probably not popular (proj)

These two thresholds very quickly changed the file size, as you can see in the interactive. Important calls/attr dropped appear below.

UPDATE:

Using path filtering and a bit of over elimination (only considering unique paths), I got the project threshold to work better, and so reduce the size of p=3 r=2 by 3/4

Result

There are many ways to use this table, you can look at the pyhash repo. I published prebuilt datasets in releases.

My ty fork includes a table (raw threshold 7), hopefully it will be merged to ty sometime. In the meantime, you can build it and use it manually

This was one of the most interesting projects I’ve done. Looking at vscode from the viewpoint of a new editor developer presented the opportunity to do this.

“In all affairs, it’s a healthy thing now and then to hang a question mark on the things you have long taken for granted” ~ Russell

minimal research

Just the other day, I was browsing the homepages of big companies (with the purpose of finding interesting things) and was shocked by the Intel.com website.

The first thing that got me interested, is that from a simple search of debug in the developer tools I get a result like location.search.includes("debugger") (check if the URL part after ? include debugger). That is weird, but not uncommon for a company to have. I append ?debugger=true and noticed the page didn’t change much.

But after comparing to the search result for location.href= I discovered the page did change (in fact, it would have on any URL parameter), because now there are more JavaScript scripts loaded: most notably, there is now a new file commons-page.min.js which includes most of the results (which is already a red alert, as setting location.href to non-fixed addresses is not recommended). The new file is non-debuggable and generated dynamically through a jQuery script so it’s not possible to debug in chromium and showed up as sourceNNN in Firefox.

Challenge

The new script uses the function CQ.shared.XSS.getXSSValue to protect against XSS. Here is the function definition:

shared.XSS.getXSSValue: function (a) { 
    return  _g.Util.htmlEncode(a)
}
Util.htmlEncode: function (a) {
    return String(a)
        .replace(/&/g, '&')
        .replace(/>/g, '>')
        .replace(/</g, '&lt;')
        .replace(/"/g, '&quot;') 
}

The function escapes HTML tags (like < and > along with escaping double quotes).

As of writing this article, the XSS is live at intel.com. You can go right now to experiment and find the XSS on your own. The XSS part seems unrelated to the structure of the file, as if someone added it in a hurry.

Since I found this XSS the WAF (Akami edgesuite) became a little more sophisticated, and now blocks all obvious attempts to run XSS. As of today, when you want to test your XSS, make the target be javascript:decodeURIComponent(location.hash), and experiment with the hash instead of the URL parameters.

Go now to https://www.intel.com/content/www/us/en/homepage.html?debugger=true take 5 minutes, and try to find the XSS yourself!

Note to Brave browser users: Brave shields block the dynamic file as “fingerprinting” so disable that.

var queryParamsURL = CQ.shared.XSS.getXSSValue(window.location.search)
  , queryParams = queryParamsURL.slice(1);

if (queryParams && queryParams.includes("doRedirect") && queryParams.includes("timeDelay")) {
    for (var url, timeDelay, params = queryParams.split("\x26"), i$22 = 0; i$22 < params.length; i$22++) // "\x26" = "&"
        params[i$22].includes("doRedirect") && (url = params[i$22].split("doRedirect\x3d")[1]), // "\x3d" = "="
        params[i$22].includes("timeDelay") && (timeDelay = params[i$22].split("timeDelay\x3d")[1]);

    url && timeDelay && setTimeout(function() {
        window.location.href = url
    }, timeDelay)
};

const queryParamsURL = CQ.shared.XSS.getXSSValue(window.location.search)
const url = queryParamsURL.get("doRedirect")
const timeDelay = queryParamsURL.get("timeDelay")

if (url && timeDelay){
     // run `window.location.href = url` after timeDelay ms.
    setTimeout(()=>{
        window.location.href = url
    }, timeDelay)
}

intel

Intel doesn’t provide a bug bounty for website vulnerabilities as they don’t consider that a product. So I opened up an informative bug report, here is the response: However, they didn’t respond to any of the emails I sent to external.security.research@intel.com.

I wonder if one of the employees added it (when they were fired, for example), as the XSS doesn’t fit at all in the tracking file, and it was obfuscated strangely.

I hope you enjoy testing out intel’s new XSS :)

:visited security

My first instinct was to try and see if a site can detect if this is applied. I was far from the first one to try this, and the browser already puts quite a few security measures. For example, you can only change the colors. No other CSS properties allowed. Moreover, the color’s alpha will be kept as in the unvisited link (that means, a color like transparent will make the element fully transparent even if the :visited CSS rule says otherwise). Btw, JS API, and copy-pasting of HTML lie about :visited.

Eventually, I gave up on the site detecting which colors are displayed to the user. Instead, I started thinking how it can change the entire user experience.

Imagine a phishing website popup with two close buttons, both links to different sites, when the default color is white, and it turns black if the link is in the user browser history. (Another close button appears after some time to rescue you if you haven’t visited either of those sites) [I probably visit too many Chinese phishing websites :)]

This led me to start searching for a game to get users to tell me which sites they’ve visited.

games

I tried various types of games [mostly written by GPT] to find the best fit.

Eventually, I settled on a ‘catch the falling objects’ type of game because I have no reliable way to check if the user looses (as I rely on them to provide this information). In a game like Snake, for instance, the user would expect me to know when they eat an apple. Additionally, the accuracy is very high for both sides—there’s less chance of the user clicking on a transparent object, and I can detect if they click on an object of any color, allowing me to fake knowledge effectively.

falling history game

It took some effort to make GPT avoid using the canvas, but eventually I got to this: a random character created every N ms with the link location from a file with the most visited websites.

If the link hasn’t been visited, the character’s color matches the background. If it has been visited, the character displays in a bold color. These links fall down the screen at an increasing speed.

Each time you click on any <a> tag, your score increases. Occasionally, the link corresponds to the current site, which I know is visited. If those links go unclicked, your score decreases.

Since the <a> tag behaves unpredictably with hovering and clicking (e.g., if I want to cancel hovering and prevent the click from navigating to the href, I need to cancel all events, which also prevents capturing the click), it makes sense to wrap each <a> tag with a <div> that can be clicked.

The biggest issue was to avoid object colliding, because then when the users click the object they see, they accidentally also click the invisible element they don’t. To solve that I needed to make all elements move in the same speed, and use better random location spawner.

For the random character I’ve found Unicode range from 0x1F300 to 0x1F5FF perfect for emojis that HTML supports and can be colored.

top websites list

Every most-visited websites list that I could find used traffic or connections to websites to measure the visit level of the site. In this case, this is clearly a wrong measure, as I want to know which URL is most likely to be in the user browser history. For example googleusercontent.com homepage, while getting a lot of traffic, will not be in any browser history unless the user opens his Google profile picture in new tab, then deletes the path. Same about storage.googleapis.com. Finally, I created a list based on myself, and on compilations from other lists. It gave me the option to do a programming-focused list, and add some funny stuff, such as the StackOverflow Question of how to exit vim. And also, this important video

go try it now

From a simple look through the site, it has this URL (you can navigate there from “My account”→”New Data Source”), which lets you connect to a remote database. You have 12 choices, and three of them caught my eye:

1. OData (Beta)
2. On-premises Database
3. Cloud Database (i.e. SQL database)

All of which let the user input a URL to connect to.

So, I set up my webhook.site, and I started experimenting:

SSRF

[explanation what is ssrf here]

Endpoints

I didn’t succeed with the “Cloud Database”, but in the other two SSRF works, with almost no limitations:

• On both, the URL 127.0.0.1 [or localhost] results in Cannot assign requested address [::1].
• The odata endpoint can only be HTTPS. (“for security reasons”). Redirects are not followed.

The Odata option can result in mainly 4 states:

• SSL connection could not be established - http server [e.g. https://metadata]
• Name or service not known - invalid domains, IPs and localhost
• No connection could be made because the target machine actively refused it - closed ports. This is very slow.

I guess at this stage I could try to scan the internal network, but the API is very slow, especially when most of the IPs are going to fall into the target machine actively refused category.

The on-promise option is more blind but has even fewer limitations: it’s meant to connect with DreamFactory servers, You can use both HTTP and HTTPS. Other protocols results in asp.net error like this one:

Unable to cast object of type 'System.Net.FtpWebRequest' to type 'System.Net.HttpWebRequest'.

Information Disclose

When I sent some HTTP requests to my webhook.site, I saw some weird headers.

Turns out, if you send the same HTTP request twice, it will be requested from the proxy, in this case, envoy proxy. Due to a misconfiguration, envoy sends the headers X-envoy-peer-metadata, and x-envoy-peer-metadata-id.

The metadata-id contains sidecar~10.32.xx.xxx~appsheet-server-cxfxxxd.appsheet~appsheet.svc.cluster.local but second header (X-envoy-peer-metadata), which is base64 encoded, includes a surprising amount of information:

APP_CONTAINERS: appsheet-server
CLUSTER_ID: xxx-appsheet-prod-europe-westx-main-cluster-europe-westx
INSTANCE_IPS: 10.32.xx.xxx
ISTIO_VERSION:1.xx.xxx
app: appsheet-server
deploy.cloud.google.com/release-id:main-server-xxxx
security.istio.io/tlsMode: istio
service.istio.io/canonical-name: appsheet-server
service.istio.io/canonical-revision: latest
skaffold.dev/run-id/5/3xxxxxx
MESH_ID: proj-10xxxxxx
NAME: appsheet-server-xxxx
NAMESPACE: appsheet
gcp_gke_cluster_url: https://container.googleapis.com/v1/projects/appsheet-prod/locations/europe-westx/clusters/main-cluster-europe-westx
gcp_location: europe-westx
gcp_project: appsheet-prod
gcp_project_number: 10xxxxx
WORKLOAD_NAME: appshet-server

(I did censor the information, but at the time of writing this anyone can just get this info)

Google

I reported it to google. Sometimes you get lucky, sometimes not. This time, google closed my report with “Intended Behavior”

I hope you enjoyed the article, and enjoyed this hidden intended feature of the appsheet :)

Simple research ⇾ XSS

The site has many apps, all of them are listed inside the robots.txt file:

#apps-toolbox
User-Agent: *
Allow: /apps/main
Allow: /apps/browserinfo
Allow: /apps/checkmx
Allow: /apps/dig
Allow: /apps/har_analyzer
Allow: /apps/loganalyzer
Allow: /apps/loggershark
Allow: /apps/messageheader
Allow: /apps/recovery
Allow: /apps/useragent
Allow: /apps/other_tools
Allow: /apps/encode_decode
Allow: /apps/screen_recorder
Disallow: *

Most of the tools are accessible from the /apps/main menu, however, the recovery app (at /apps/recovery) isn’t.

From a simple search in google I see the recovery app has these sub-pages:

recovery/domain_in_use
recovery/form
recovery/ownership

All of which receive many parameters from the query string in URL (parameters in {url}?parm1=1&param2=2...): visit_id, user, domain, email and some more.

In google search I also spotted a result that has Verify that you own example.com title, with this link : https://toolbox.googleapps.com/apps/recovery/ownership?domain=example.com&email=email@example.com&case=45500368&continue=/apps/recovery/...

The server apparently just verifies that the email matches the domain, then presents a page with some thank you text and a continue button:

And the link in the continue button, was … you guessed it: just taken from the continue URL parameter.

So I tried placing there continue=javascript:alert(document.domain), and… It works!

The site didn’t use any CSP, or any protection at all. So I also could send and receive data from external sites: (e.g. continue=javascript:fetch(%27https://api.ipify.org?format=json%27).then(response=%3Eresponse.text()).then(data=%3E{alert(data);%20}), which alerts the user public ip). I reported it to Google.

Reward

Since this is an XSS, and its on a normal Google application, it falls into the 3133$ square in google rewards. Therefore, I got more than twice than I got to both parental control bypasses (googles secret browsers) combined.

I name this article “Common” because it’s really an openredirect->xss by the book. No thinking is required, just trying to change random parameters on URLs.

How to get there?

1. Enter the Contacts app - using the “emergency call” button after the normal unlock of the phone. (assuming you not are reading this blog in lock-down mode, you can just open the normal Contacts app).
2. Edit existing contact (or add new contact), then edit it, and scroll until “More fields” and click on it.
3. In the “Website” field enter this website: “https://gds.google.com/gmsdrops”.
4. Save the contact, then click on the link.
5. You should now see “Your Android device just got better” (it’s a Google lie 🙂). Click “Show me”.
6. Click “Learn more”. If you don’t have that, click “next” until you have it.
7. Now you are in the browser. Resize it by moving it up. Click the hamburger menu, then click the big “Google Help” text.
8. Click the hamburger menu again. This time just click “Google”.
9. You may or may not be already signed in to this browser. If you are signed in, you can log out from Google. It does not affect your Chrome browser. There you have it. A full untraceable browser inside the parental lock-down mode!

Why does it work?

In lock-down mode, google “locks” all apps (including the android launcher and parts of the system) apart from “Google Play Services” (which is used to display the popup message and enforce restrictions) and the Contacts app (for phone). As last time, It’s still the fault of the same app: Google play services. https://gds.google.com/gmsdrops is a deeplink to the Android “what’s new”. (you can also open it from here, and if your browser forwards deeplinks you probably get a message asking you if you want to continue to external app/google play). While parental control doesn’t allow you to open deeplinks, it does allow the Contacts app to do so. When you click on the website field of a Contact, it’s the Contact app which opens the link. So it’s not blocked.

Screen pinning bypass

android (11+) has an Android screen pinning feature, which basically make it possible to give your phone to someone, open on a specific app, and prevent the user to move to another without your permission. I haven’t done research on that, but I believe the most popular use-case is when you give your phone to someone to make a phone call. This time we cannot use the same link as before, as screen-pinning prevents opening new apps, and the previous link opens the “Google Play Services” app. But we can use another deeplink which is managed by the same app: Google Podcast. It’s possible because this deeplink is opened as a popup window instead of a full app.

1. Add website to contact in the same way as before. Enter the website “https://podcasts.google.com”
2. Click the link when the app is pinned.
3. You should now see the Google podcasts popup window. Click on the big icon of your Google account, then click “Content policies”. Now you are in the default browser. The exact place where you should not be when someone gives you their phone to call. For breakthrough use the same instructions as before:
4. Click the hamburger menu, then click the big “Google Help” text.
5. Click the hamburger menu again. This time just click “Google”. You got it. A complete bypass.

Google Response

I reported it to Google, as two different cases : one for parental control bypass, and another one for android screen pinning bypass. They merged the parental one into the screen pinning bypass one, then they managed to “forget” about the duplicate cases. This is the response I’ve on the screen bypassing case (because of course screen bypassing and parental controls is intended to be bypassed):

and this confusing response about the duplication:

I hope you enjoy your secret untraceable browser.

It happens a lot that I need a media converter: for example, to convert mp4 to GIF, that I can send on WhatsApp.

What most people do is just search google for “mp4 to GIF online converter” and get a lot of sites that are either full of trackers and keep your data, or force you to create an account.

But the real problem began after I had a larger mp4, then most converters wouldn’t even let me upload, and those who do, have a very long upload time. So I was forced to use ffmpeg (FF MPEG command-line program to convert media), which is not a bad program, it’s just unintuitive (or has unintuitive defaults) to the level that instead of searching for an online converter I was searching “FFmpeg command mp4 to GIF” (the default is ffmpeg -i inp.mp4 out.gif, which just puts each frame of the mp4 inside the GIF, so for 3.6 mb mp4 file I get 74 mb GIF …)

So I built my own online converter.

building the converter

why serverless

There are two approaches for conversion websites:

1. The client upload the files to the server, it runs FFmpeg on each of them in a safe and orderly manner, and returns the output.

2. The client uploads the files to the javascript on the website, and the JavaScript converts the files locally in the browser of the client. In this case, the server is used just to serve the HTML and JavaScript file (serverless).

In most cases of conversion, the first approach is the only approach that is possible, and this is the approach you see in google results.

Since I moved recently from WordPress to jekyll (from full server to serverless), as you can guess, I choose the serverless approach.

FFmpeg

There is something special regarding media conversion sites: most of them use the same open source tool, just with different frontend and options. This awesome tool is called FFmpeg (or Fast Forward Moving Picture Experts Group), and it’s written in c.

But there is one problem. How to run this serverless? You can’t just run c code on a browser and expect it to work. But with WebAssembly, it’s possible. Still, the FFmpeg code is not really WAsm-compatible.

But someone ported FFmpeg (using Emscripten) to WebAssembly, and created the awesome ffmpeg.wasm and even created a JavaScript/typescript interface for it.

So, after we know how FFmpeg can run in the client browser, let’s get started.

start the project

spoiler: the converter is online at this site at /private-convert

Let’s start the project. I want to leverage the typescript interface that ffmpeg.wasm has, so I use create-react-app from Facebook:

yarn create react-app my-converter --template typescript

The first thing you need is an interface. I don’t really like designing everything from scratch, so I used ChatGPT/Bard to write a simple interface. They wrote a lot of code (you can look at it in this commit), but here is the important code:

enum Screen {
  UPLOAD,
  PREVIEW,
  CONVERTING,
  CONVERTED
}
const App: React.FC = () => {
  const [currentScreen, setCurrentScreen] = useState<Screen>(Screen.UPLOAD);
  const [selectedFiles, setSelectedFiles] = useState<FileList | null>(null);
  const [convertedFile, setConvertedFile] = useState<File | null>(null);
  const [conversionProgress, setConversionProgress] = useState<number>(0);
...
  const handleReset = () => {
    setSelectedFiles(null);
    setConvertedFile(null);
    setConversionProgress(0);
    setCurrentScreen(Screen.UPLOAD);
  };
  const handleConvert = () => {
    setCurrentScreen(Screen.CONVERTING);
    simulateConversion(); // Placeholder for actual conversion logic
  };
  const renderScreen = () => {
    switch (currentScreen) {
      case Screen.UPLOAD:
        return (<input type="file" multiple onChange={handleFileUpload} />)
      case Screen.PREVIEW:
        return (
          <div className="content">
            {/* Display preview of uploaded files */}
            <select className="dropdown">
              {/* Dropdown for "convert to" options */}
            </select>
            <button className="action-button" onClick={handleReset}>Reset</button>
            <button className="action-button" onClick={handleConvert}>Convert</button>
        )
      case Screen.CONVERTING:
        return (<p>Converting...</p>)
      case Screen.CONVERTED:
        return (<button className="action-button">Download</button>)

The code defines 4 states the site can be on:

1. upload - where the user uploads the file

2. preview - where the user chooses which format they want as a target

3. processing (converting) - a ProgressBar while the file is being processed

4. download (converted) - where the user can download the converted files.

Great. Now we have an interface, and we can continue to the interesting part : the conversion.

The Conversion

There are two versions of ffmpeg.wasm, the multi-thread (called @ffmpeg/core-mt) and the single-threaded (called @ffmpeg/core). Sometimes Chromium-based browsers do not support multi-thread. The single-threaded works on all browsers, but it’s much slower compared to the multithreaded version. So let’s load the multithreaded version only on Firefox. Here is the load function: (this file is also here)

import { FFmpeg as FFmpegCore } from "@ffmpeg/ffmpeg";
import { toBlobURL, fetchFile } from "@ffmpeg/util";
class ffmpegCls {
...
  async load(): Promise<void> {
    let is_firefox = navigator.userAgent.toLowerCase().includes('firefox');
    let core_path = is_firefox ? "core-mt" : "core"
    const base_url =  `https://unpkg.com/@ffmpeg/${core_path}@0.12.2/dist/umd`
    console.log("loading ffmpeg from",core_path,"is_firefox:",is_firefox)
    let coreblob = await toBlobURL(
      `${base_url}/ffmpeg-core.js`,
      "text/javascript",
    )
    let wasmblob = await toBlobURL(
      `${base_url}/ffmpeg-core.wasm`,
      "application/wasm",
    )
...
    await this.ffmpeg.load({
      coreURL: coreblob,
      wasmURL: wasmblob,
      workerURL: workerblob
    });
    this.loaded = true;

  }

Which loads the correct WAsm version based on the browser.

Let’s also define a function to run the FFmpeg command:

  async exec(inputFileName: string, OutputMimeType: string, inputBlob: string, outputFile: string, args: string[]): Promise<File> {
...
    await this.ffmpeg.writeFile(inputFileName, await fetchFile(inputBlob));
    const commandList = ["-hide_banner", "-i", inputFileName, ...args, outputFile].filter(el => (el !== '')) // remove empty strings
    console.log(`running ffmpeg command [${commandList}]`)
    await this.ffmpeg.exec(commandList);

    const data = await this.ffmpeg.readFile(outputFile);
    const blob = new Blob([data], { type: OutputMimeType });
    return new File([blob], outputFile, { type: OutputMimeType })
  }  

Good. Now we have class to load FFmpeg, and a function that runs commands on an input file and gets an output file. But the exec function we defined takes some parameters we don’t know:

1. inputFIleName,inputBlob, OutputFile - we know. The input is what the user supplied and the output just replaces the extension.

2. OutputMimeType - we need a map {ext_format: mimetype}

3. args - we need a map {conversion_type: args}

OutputMimeType is necessary so that the browser knows how to save and display it, and args is necessary to change the default behavior of FFmpeg because otherwise we will have things like 74 mb GIF from 3.6 mb MP4.

The map

Let’s define the convert option:

export interface ConvertOption {
  extension: string; // for example "mp4"
  mimetype: string; // for example "video/mp4"
  full_string: string; // for example "MPEG-4 Video"
  optional_convert_routes: ConvertRoutes; // for example: {mkv: ["-vcodec","copy"]}
}

then, we can define the “standard/normal FFmpeg args” for most video formats:

interface ConvertRoutes {
  [format: string]: string[];
}
const normalVideoRoutes_video: ConvertRoutes = {
  mp4: [],
  mkv: ["-vcodec","copy"],
  avi: [],
};

and finally write the map (you can look on the full map at this file):

interface ConvertOptionsType {
  [extension: string]: ConvertOption;
}
export const ConvertOptions: ConvertOptionsType = {
  // -- video
  mp4: {
    extension: "mp4",
    mimetype: "video/mp4",
    full_string: "MPEG-4 Video",
    optional_convert_routes: normalVideoRoutes,
  },
  mkv: {
    extension: "mkv",
    mimetype: "video/matroska",
    full_string: "Matroska Video",
    optional_convert_routes: copyWith(normalVideoRoutes, {
      mp4: ["-codec","copy"],
    }),
  },

I added some more features (for example, multi-files, FFmpeg logs). You can look at the final App.tsx on my GitHub, or just enjoy it right now: online at /private-convert

I hope you enjoy it and if you have any Idea how to make it better, let me know in the comment section, or just with issue or pull request to the private-convert GitHub Repo.

backstory

I read hackernews sometimes, and articles like I mirrored all the code from PyPI to GitHub and analysed it made me think, maybe there are less obvious places with leaked information.

This was in the back of my mind until recently, when I tried to contribute to the buildozer project. I implemented rich logs, and I felt like I understood buildozer, so I searched for stackoverflow questions about it. Then I discovered that buildozer, unlike beeware , actually dumps the full environment when a command fails (for example output, see this old question), and thought “what if this user built the app with a sensitive api ?”

parsing

So, I downloaded stackOverflow from the stackexchange archive.org dump, and started to think how to parse it. It’s a huge (103G) XML file, where each line is a question or answer.

I tried different “leak detect” tools, most of them crashed or taking too much cpu:

• gitleaks : fatal error: runtime: out of memory (Update: After I posted this article, @zricethezav from GitHub made a gitleaks PR to fix this, and now (v8.18.1) it does not crash.)

• truffleHog : Actually works (and takes 100% cpu), but giving very poor results (e.g. a simple %s is considered a SQL server base64 encoded: Detector Type: SQLServer,Decoder Type: BASE64,Raw result: %s ).

• ripsecrets : no output 1 hour after I run it.

• (Yelp) detect-secrets: Traceback (most recent call last): ... MemoryError

• ripgrep : freeze my system after few minutes.

So I ended up returning to my zsh and doing grep , and even writing a my own rust script to do the searches.

results

As I suspected , there are a lot of leaks in stackoverflow (on the graph, only unique and not junk data is displayed. click on a label to hide it ):

As you can see, a lot of data.

Then, I asked myself, what could an attacker do with this information? Turns out, most of it is useless:

For using most data, you need more information than just the api key. For example, for stripe , you need the customer ID. For grafana, an instance url. For aws, a site url.

And even if you have an api key which goes to a centralized location without need for a “username”, most of the data is old. All the JWTs (JSON Web Tokens) - forget about them, the average life of a JWT is a month.

Until I run a simple scan (again using the best hacking tools : xargs and curl ) against all the 74 real looking GitHub user tokens (which is a token that grants access to pretty much the full GitHub user) and discovered that 6 of them are actually valid.

Still, only 2 of them actually have bio and email, but one of them (a c/c++ developer) has a repo with 3.4k stars. So I finally found the path an attacker would take.

I sent both developers an email (I told them about this research and referred them to the question where they leak the token, with a little “if you find this message helpful, you can buy me a coffee” at the end) and they both revoked the tokens (and both actually bought me a coffee, the first time I got money since I opened this buymeacoffee account in March 2021!).

I obviously couldn’t verify all the secrets. From most of them I’ll probably be banned, so I stooped here.

cause

Unlike PyPi and GitHub leaks articles, this article is not because of people leaving the password in their deploy-to-server.py and accidentally committing it (well, sometimes they copy deploy-to-server-example.py into stackoverflow and forget to mask the id …).

Most leaks are in the output of tools, a long output that people like to copy/paste right into stackoverflow, without actually looking at it, because of people publish the output/ the tool config file of the tool they using (e.g. did you know that curl -v also displays your request with the headers, or that a long package.json with private dependencies can contain git+your-private-gh-token ? )

why is this not working?  I run this command curl -v --path-as-is 'https://matan-h.com/[redundant]'
and I get this output:
*   Trying 185.199.108.153:80...
* Connected to matan-h.com (185.199.108.153) port 80 (#0)
> GET /404/../ddebug/../my-windows-shell/../list-of-online-converter-tools/../exec_python_code_super_secret_4h0a4b?code=print("hi") HTTP/1.1
> Host: matan-h.com
> User-Agent: curl/40.4.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br
>
< HTTP/1.1 301 Moved Permanently

I hope you enjoyed the article, and pay more attention to what you copy/paste in StackOverflow.

Plugins

I didn’t like the plugin ecosystem of WordPress. I want these things (which I think should be pretty much built in):

• Insert HTML code (with script tags) to my pages (and have HTML highlighting while I edit them)
• have a contact form.
• Have a custom 404 page
• have a redirect page
• add syntax highlighting to my code blocks.
• Change my home page
• have an opengraph meta tags and sitemap.xml for SEO.

That’s right : to do these 7 basic things I had to install 7 different plugins

1. HTML Editor Syntax Highlighter -for creating HTML pages

2. WPForms - to have a contact form

3. Smart Custom 404 error page - for a custom HTML 404 page

4. Redirect URL to Post - for, well, redirect a URL to different post

5. Code Syntax Block - for highlighting in my code blocks.

6. Header Footer Code Manager - unfortunately, my theme wasn’t the kind that could be customized (at least if you didn’t buy the pro version), so I used a JavaScript code snippet to change my homepage as a sort of hacky method.

7. The SEO Framework - for opengraph and sitemap.xml

This alone was annoying. And from these 7 plugins, 4 (WPForms, Header Footer Code Manager, The SEO Framework) have a pro or a paid extension, that added banners and emails to convince me to upgrade.

But hey, there is another method instead of plugins - the raw PHP method:

The raw PHP method

PHP is a terrible language - especially for security. I recently covered a real website (written in PHP using a tutorial), which has an include statement to include txt files. The website was hacked in 2021, and the author add a filter to block including files which do not have “.txt” or start with “http”. In most languages it would be a pretty good filter. But PHP is different - it has a php://filter URL which allows PHP text to be injected.

I don’t like PHP. I always fear I’ll break something.

Security

On WordPress, security is a real issue - I needed to make sure I used the most up-to-date WordPress and up-to-date and trusted plugins, and even then, new vulnerabilities are discovered every month (mostly in plugins and WordPress PHP files), so I have to update it at least once a month.

In a static site, security is almost guaranteed : since there are no dynamic files running on the server, the only thing a hacker can do is to read your HTML and images files. (yes, that means the hacker can read your keys if you store there your secret keys)

🤫 id:AKIAIOSFODNN7TMATANH

When you store keys in your HTML, you need to make sure it’s safe to show the key to the user.

Rethinking what I enjoy on WordPress.

I have a WordPress blog - that mean, I enjoy the WordPress great markdown-like editor, I enjoy the emails from the Comment section and Contact form (although, not when they only contain please upgrade to pro).

But what is the main advantage of WordPress - databases, and user management. Since it’s my personal blog, I don’t use these features.

Since most of my blog is articles, I do not need dynamic pages as my posts are not changing while you look at them (well, apart from my 404 page). So I can use a static site and get a three times faster site without PHP code which I’m afraid to touch.

Static site challenges

Jekyll is not the only choice for static site generator. There’s also Hugo, Zola, and you can also build a static site with react If you want. But Jekyll and Zola are the only ones who really built for blogs, and Zola … well, it was too much HTML to write by hand, and I admit I don’t like the idea of a single binary that can’t be expanded by plugins.

So first I needed to find a theme : just like WordPress, Jekyll also has lots of themes where the most popular (the awesome minimal mistakes) is the one I use right now.

In my opinion, it looks pretty good by default. But there are two things I didn’t like: the homepage and the sitemap.

The homepage

by default, the homepage is just a list of the Recent Posts:

I wanted featured images, so what I really want is a gallery, not a list. To solve this I created a liquid-based page, to generate a gallery based on the posts I have.

liquid is Jekyll template language: it “runs” one time, at the build step to generate the static site. Since it runs only once for the site, and not once for each user, it is inherently safe. In liquid, I can do for loops on the posts/pages/files, if statements, and so on.

So, the code to create the gallery was (you can see the full home page in my GitHub):

<div class='gallery'>
    {% for post in site.posts %}
    <a class="gallery-item" 
    href="{{ post.url }}"
    style="background-image: url('{{ post.image }}')">

        <div class='card-content'>
            <h2>{{ post.title }}</h2>
            <p class="card-date">{{ post.date | date: "%B %d, %Y" }}</p>
        </div>
    </a>
    {% endfor %}
</div>

and now, it looks like this:

The sitemap

Most people do not care about sitemaps in sites. It looks like an old, maybe even a robots-only way to navigate site. Modern websites should already have menu with all pages, and homepage with all the posts, right?

But efficiency is the key here. If you want to look at my posts, probably the homepage wins. But if you want to navigate fast to a page in my site, I think that the HTML sitemap wins.

The default HTML sitemap is in exactly the same spirit as the homepage: just a list of pages, then a list of posts.

While it possible that some person would accidentally use it, I don’t think it would be a great experience.

(I left one of these HTML sitemaps on my site, so that robots think I’m making the site accessible to people. Don’t tell them that a sane person wouldn’t use it)

But what if there was a convenient way to see the pages and posts like a row of blocks?

For this, I built a Visual Sitemap in Liquid that does exactly this: displays my posts and pages as blocks. I used mattbrailsford/css-sitemap (well, I also modified it a bit) to make an HTML list that looks like blog. Here is part of the code:

<nav class="primary">
    <ul>
        {% for post in site.posts %}
        <li id="{{ post.slug }}">
            <a href="{{ post.url }}">
                <i></i> {{ post.title }} <small>{{ post.excerpt | markdownify | strip_html | truncate: 160}}</small>
            </a>
        </li>
        {% endfor %}
    </ul>
</nav>

The list looks like this without mattbrailsford/css-sitemap:

and the list looks like this with mattbrailsford/css-sitemap:

Well, I lied. This is how it would look with the original mattbrailsford/css-sitemap:

As you can see, I modified it a bit.

And now I really enjoy Jekyll: I love the fact I could do transparent text easily in this article (did you spot that?). I’m using MarkText markdown editor right now, so I can copy-paste pictures, and it automatically puts them in my Assets folder.

Do you have a Jekyll blog (or another static site generator) ?

Do you have a WordPress blog?

Are you satisfied with it?

Let me know in the comment section.

This has a few steps.

1. Install and setup Termux – the best terminal emulator for Android. This will run your code such as Python or Node.js.
2. Get a file manager for Termux – a graphic user interface for easier use and manage your Termux files
3. Get a graphical file editor – sometimes, it’s better to edit your project with an actual GUI editor
4. Install AndroidIDE for coding Android apps from an Android phone – if you want to develop an Android project. (optional)

Install Termux and basic Termux utility:

Install Termux from f-droid or from GitHub (google play is no longer an option). I recommend the following setup [paste in Termux is a long click on the screen and click paste]:

First, write the command termux-setup-storage – that will allow Termux to read and write (but not execute!) files from the shared storage (/sdcard).

Second, let’s install some utilities that I think every development environment should have (or at least these are the utilities I always use) :

pkg install git python micro zoxide fd sd ripgrep

Here is an explanation for each of the tools:

• Git – for clone, pull, and maybe even publishing projects
• Python– a simple and intuitive programming language. I find this very helpful even when I coded an Android app
• Micro – the best editor for Termux that I know :
  When I edit something in Termux, I need a simple and intuitive text editor. I cannot waste keystrokes by using an editor that’s built for a computer keyboard, and uses numbers and symbols a lot. In Android (with most keyboards) the symbols are 2 buttons away, but control is in the Termux keyboard anyway. So I wanted a text editor that has one mode, and uses simple control shortcuts. Micro-editor is exactly that: it uses ctrl+s to save, and ctrl+z is to undo, etc.
• zoxide – smarter cd command. It keeps track of the directories you use the most, so if you type z myproject, it will remember that it is located in the /sdcard/termux/projects/myproject, for instance.
• fd – a simpler find command, that uses fd regx to find all files containing regx, and fd -e html to find all files with extension html
• sd – (far) better than sed. Find and replace regex just by sd before after, for example, to replace newlines with commas: sd '\n' ',' (here is how to do it with sed)
• Ripgrep – fast search for regex in all files in a directory (for example, rg javascript: to find all files that has the string “javascript:” in them)

Third, I really cannot work without powerlevel10 (or oh-my-zsh, but that’s more like a framework instead of a theme. I still use it in my Termux, since my config files for Linux also use it), zsh-autosuggestion and zsh-syntax-highlighting plugins, so let’s install them:
Install zsh using pkg install zsh, and then use chsh -s zsh to make it the default, then clone the plugins

git clone --depth 1 https://github.com/zsh-users/zsh-autosuggestions ~/.zsh/zsh-autosuggestions
git clone --depth 1 https://github.com/zsh-users/zsh-syntax-highlighting.git ~/.zsh/zsh-syntax-highlighting
git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ~/.zsh/powerlevel10k

Then edit ~/.zshrc (the zsh config file) using micro: micro ~/.zshrc then write:

source ~/.zsh/zsh-autosuggestions/zsh-autosuggestions.zsh

source ~/.zsh/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

source ~/.zsh/powerlevel10k/powerlevel10k.zsh-theme
eval "$(zoxide init zsh)" 

Hit ctrl+s to save, then ctrl+q to quit micro.

Write the command exec zsh (which reload the shell). Powerlevel10k Configuration wizard should come up, and ask you some questions: for the first question: yes, you want to install the recommended font. Otherwise, you will not see your theme nicely. Answer the other questions with you liking your shell to be. For the last question: yes, you want to modify your .zshrc

After this, you are done setting up Termux (although I would recommend you add some aliases to your .zshrc such as alias l='ls' to make the letter l do the same as ls)

Get a file manager for Termux

Unless you want to manage your Termux files using a terminal file manager like nnn, you would probably be more comfortable using a graphic file manager.

There are only two file managers I know that can edit Termux files: the built-in (hidden) files app and material files, and they both require some steps to set up:

The hidden files app (the screenshot on the left): the simplest way to access it is to install this shortcut app (it’s not open source, probably because it’s only a simple shortcut app, so it has very little code) created by Marc apps & software. If you have activity manager installed, you can search for files and discover you have a package called something like com.google.android.documentsui. You can create a shortcut to the activity FilesActivity.

Both ways give you access to the Android built-in file manager, that will allow you not just to manage your Termux files, but also to manage android/data folder which you can’t access with material files.

Material files (the screenshot on the right): install this from the play store, then click the hamburger menu, click add storage..., External storage. Click the hamburger menu again and click Termux then click use this folder.

Graphical text editors for full projects

If you’re developing something like a React website, using the micro editor to edit individual files is simply not enough.
For those types of projects I use Acode which is like vscode to edit files while viewing the whole project.

Complete IDE for coding Android on Android

Setting up Android development using Termux and Acode is very hard (you need to install openjdk, install Gradle, install Android SDK, use a template to create new app…) and the sync/compile process is complicated (gradle build,⁣somehow get Gradle to sync without build, etc.). Fortunately, there is an open source app called AndroidIDE that does the things Android-studio does: sync Gradle in the background, view files in a convenient way, a run button, and even a built-in Termux (I am not kidding, the app has a full Termux inside).

To install it, follow the official installation docs. This is the only app I have that is not in any app store. And like most of the apps in this blog post, it is open source. However, it is not on f-droid, and you have to manually install it from an APK file.

So, next time you’re armed with just your Android and a good idea for a project, remember, turning it into reality is just a few taps away. Happy coding with your pocket dev environment!