Tokenizer

Articles about Tokenizer

1. Article · AI Agents

   Why There Is No "Tokenization-Only Model" in Modern NLP

   Uncover why tokenization in NLP isn't a neural model like embeddings. Explore its deterministic nature, algorithm-based approach, and tight coupling with LLMs.

   Jayant KumarJan 62,3521,000
2. 
   Article · .NET

   Interpreter Pattern in .NET

   The Interpreter Pattern in .NET facilitates language interpretation by defining grammar and providing a framework to interpret expressions. It's ideal for implementing domain-specific languages or complex rule-based systems with ease and extensibility.

   Jean PaulMay 8, 202437,0491,000
   
3. Article · Current Affairs

   C# Tokenizer

   A simple tokenizer in csharp without using regex or MatchCollections.

   GSgary sunSep 5, 202368,0931,000
4. Article · Angular

   Email Tokenizer For AngularJS Directive

   This article explains the AngularJS directive for valid email tokenizer.

   Muhammad HassanApr 6, 20155,9621,000
5. 
   Article · C# Strings

   C# Versions of Java's StringTokernizer

   Here is another C# version of the Java StringTokenizer Class from the version posted August last year on your site. Given a string and a set of delimiters, the class return an enumeration of substrings based on the delimeters passed in.

   May 31, 200715,3831,000