Encontrar

USER>:sql
SQL Command Line Shell
----------------------------------------------------
 
The command prefix is currently set to: <<nothing>>.
Enter <command>, 'q' to quit, '?' for help.
[SQL]USER>>set transaction isolation level read committed    // Read Committed セット
:
[SQL]USER>>select Name from Sample.Person where Age < 10
2.      select Name from Sample.Person where Age < 10
Name
 
 
[SQLCODE: <-114>:<ひとつまたはそれ以上のマッチする行が別のユーザによりロックされています>]
[%msg: <Unable to acquire shared lock on table Sample.Person for RowID value: 4>]
0 Rows(s) Affected

こちらは、

他のプロセスが該当テーブルの該当IDをロックしている
ロック閾値によるテーブルロックを行っている
システムのロックテーブルが不足しているためのロックテーブルフル状態

のいずれかになっているものと考えられます。

ロックテーブルフルの場合、messages.log に「LOCK TABLE FULL!!!」のエラーが記録されます。

ロックテーブルフルの場合にロックテーブルサイズを変更する方法は、2023.1 より前のバージョンの場合はこちらの記事をご覧ください。

2023.1以降のバージョンでは、ロックテーブルサイズは既定で 0 に設定されています。
0 の場合は、構成されたグローバルバッファサイズ（データベースキャッシュサイズ）に基づき、最も効果的な設定値が自動構成されます。

【ご参考】
SQLのUPDATEやDELETEで、ある程度の数のレコードを一括更新するとSQLエラー -110 が発生します

Comentarios (0)0

Inicie sesión o regístrese para continuar

Artículo

Alex Alcivar · 28 jul, 2024 Lectura de 6 min

Open Exchange

#API #CSS #Framework #Frontend #GitHub #JavaScript #JSON #Python #InterSystems IRIS

For a long time I have wanted to learn the Django framework, but another more pressing project has always taken priority. Like many developers, I use python when it comes to machine learning, but when I first learned web programming PHP was still enjoying primacy, and so when it was time for me to pick up a new complicated framework for creating web applications to publish my machine learning work, I still turned to PHP. For a while I have been using a framework called Laravel to build my websites, and this PHP framework introduced me to the modern Model-View-Controller pattern of web programming. To make things even more complicated, I like to build my front-ends with a modern javascript framework. I am most familiar with Vue.js so for this project I stuck with that even though it is more common to hear of people using React.

Why use a complicated framework anyway? What are the biggest challenges to learning a framework like Django, Laravel, React, or Vue?

Everyone has their own answer, but I have grown to love MVC frameworks because they provide so much guidance on how to structure your app. It prevents me from needing to reinvent the wheel every time. At first these framework can seem constraining and cryptic, but once I become familiar with the structure, I find it is easier to add new features.

The challenge is that things can become almost too simple. Frameworks like Django rely on a lot of shorthand or assumptions that might be based on a familiar concept, but have a particular unfamiliar name and structure in Django. In my application, Django handles the APIs and all the web routing. If I want to add a new API endpoint, I need to add a function to a file in my views.py, then go to the urls.py file and add a statement to import that function and another statement to define the URL where that API endpoint is available. After that, I need to edit my front-end Vue component to use javascript to query that endpoint to obtain data and show it or manipulate it for the user.

Once my project is set up, adding functionality like this is fast. I only need to add about four lines of code, and then I can concentrate on the logic needed in my new function in the views.py file to process the HTTP request and return the needed data in JSON format. The challenging part is learning what those files are and how they work together to create a whole application.

I find the best way to learn a framework like Django is to find a working example and start trying to make small changes to get a feel for the flow of data. Refer to the documentation as concepts start to become more clear and make more sense. Ask AI models to explain pieces of code and what the different standard files in a framework will do. It doesn't take long to realize that these tools emerged as ways to save time in the long run and make it easy to maintain and update your application. Since the Django and the Vue frameworks have a standard structure, when you come back to them later you will find it is easier to understand why you coded things a certain way, and easier to re-establish familiarity with your work. It's also easier to pick up someone else's application and understand the core functionality since you are familiar with the basic structure of their application.

So what are some fundamentals of Django that would help someone getting started? For me the first thing to understand is that Django projects are generated by running a command to create a new django project, and this generates a set of basic files and folders that constitute a "base project" that you can use to start building. The project folder will have several python files with settings that apply to the whole project. The important ones you will find yourself visiting frequently are settings.py, where all your settings are, and urls.py. When you have a question like "How does Django decide where to put my static files", the answer is usually somewhere in the settings.py. When you want to add a new URL to your application, you will need to update the urls.py file.

Along with these project level files, you then create a folder for each app in your project. These apps then need to be registered, i.e. named, in the settings.py file. The main app folder in my project is called documents. Inside that folder I have a models.py file, a serializer.py file, a views.py file. There are others, but these are the important three.

Inside models.py I specify my Document object and the fields it has. Django does the work for me of creating a Documents table in the IRIS database with the schema necessary to save the information I plan to store in the Document objects. In my models.py file I tell it that my Documents all have a name, which is a character string no more than 255 characters, a content field, which is just a large amount of text, as well as a database name where the vectors are stored (another text field), the embedding type (another text field), and finally the vector embedding dimension, which is a number. Using these definitions, Django creates the necessary database table with the necessary column types. Saving objects to the database is then as easy as Document.save().

Inside the serializer.py file is simply a definition on how to convert your object into JSON and vice versa. For basic use cases there is a standard way to define it, and you can see it in this project.

Now we get to the point of Django, the views.py file. This is where we define the functions that accept HTTP requests and return data like an entire HTTP response, or a JSON response in the case of a JSON API. This means Django can deliver an entire web page and also be the front-end of your app, or it can just provide JSON data, and you can build your front-end on an entirely different platform.

At first having the use all of these seemingly arbitrary files and conventions can feel like a chore, but once you see that doing so means your application just starts to work and process HTTP requests and provide the correct data in response, it can be very fun to keep building new features and capabilities. One you've define one object, web route, and function to handle an HTTP request, you realize how easily you can define a second and third to add functionality to your application.

I forked my project from the Iris Django Template create by @Guillaume Rongier here on github: https://github.com/grongierisc/iris-django-template

That template contains only Django, and it was extremely helpful for learning the Django framework. One of the major additions I made was to add Vue.js with Tailwind CSS to show that you can integrate a modern Javascript framework with this package and have a Single Page Application running on IRIS. A Single Page Application is a javascript application that send xhr requests to get JSON data and updates the page dynamically without ever fully reloading it. It has its pros and cons, but it is a hallmark of modern web development.

I encourage people to look at my project as not just en example of RAG and Vector Stores on IRIS, but also as a template for using Django with Vue.js and Tailwind to easily and quickly create modern flexible web application on top of IRIS. That repository is on Github here: https://github.com/mindfulcoder49/iris-django-template

I'd be happy to answer any questions or provide any insight I can into any issues anyone might run into trying to adapt this project for their own use.

2 comentarios

Comentarios (2)1

Inicie sesión o regístrese para continuar

Pregunta

biubiu biubiu · 27 jul, 2024

#XML #InterSystems IRIS

Implement a function, write a program, and convert any XML-formatted string into a %XML.Adaptor object without having to create a Moudle class. This makes it easier to work with XML data. How to write it；

Comentarios (0)1

Inicie sesión o regístrese para continuar

Artículo

Alex Alcivar · 27 jul, 2024 Lectura de 7 min

Open Exchange

#Artificial Intelligence (AI) #ChatGPT #CSS #Frontend #Generative AI (GenAI) #Interoperabilidad #JavaScript #Python #Vector Search #InterSystems IRIS

I received some really excellent feedback from a community member on my submission to the Python 2024 contest. I hope its okay if I repost it here:

you build a container more than 5 times the size of pure IRIS

and this takes time

container start is also slow but completes

backend is accessible as described

a production is hanging around

frontend reacts

I fail to understand what is intended to show

the explanation is meant for experts other than me

The submission is here: https://openexchange.intersystems.com/package/IRIS-RAG-App

I really appreciate this feedback, not the least because it is a great prompt for an article about the project. This project includes fairly comprehensive documentation, but it does assume a familiarity with vector embeddings, RAG pipelines, and LLM text generation, as well as python and certain popular python libraries, like LLamaIndex.

This article, written completely without AI, is meant to be an attempt at an explanation of those things and how they fit together in this project to demonstrate the RAG workflow on IRIS.

The container is large because the library dependencies needed for the python packages involved in creating vector embeddings are very large. It is possible that through more selective imports, the size could be cut down considerably.

It does take time to initially build the container, but once you have done so, it takes less time to start it. The startup time could still definitely be improved. The main reason startup takes so much time is that the entrypoint.sh was updated with the assumption that changes might have been made to any part of the application since that last startup, including the database migrations, css configurations, javascript configurations, and the python backend code, and it recompiles the entire project every time it starts up. This is to make it easier to get started with developing on this project, since otherwise it can be tricky to properly run the frontend and backend builds whenever changes are made. This way, if you change any of the code in the project, you just need to restart the container, maybe recover the production in the backend, and your changes should be reflected in the interface and operation of the application.

I am fairly sure that the production in the backend is what is passing the http requests to the Django application, and is crucial to the interoperability in this package. I am new to the IRIS platform, however, and have more to learn about productions.

Next I’d like to provide a comprehensive explanation of vector embeddings, LLMs, and RAG. The first of these to be invented was the vector embedding. First we can describe a vector. In most contexts a vector is a direction. It’s an arrow pointing somewhere in space. More formally, a vector is “a quantity having direction as well as magnitude”. This could be exemplified by a firework, which travels in a particular direction and explodes at a particular point in space. Let’s say every firework is fired from the same central point, a point of origin, [0,0,0], but they all fly out and explode in a cloud around that origin point. Mathematically you could describe the location of each firework explosion using a three coordinate system, [x,y,z] and that would be a “vector embedding” for a firework explosion. If you took lots of video of a firework display and recorded all of the firework explosions as a dataset, then you would be creating a kind of vector embedding database, or vector store, of the fireworks display.

What could you do with that information about the fireworks display? If I pointed out a particular firework and asked for the fireworks that exploded closest to the same point throughout the entire display, you could find those other fireworks that exploded at nearby points in space. You just find the ones that are closest, and there’s math to do that.

Remember, we only recorded three numbers for each firework, the x, y, and z coordinates in a three dimensional space with [0,0,0] being the firework launcher on the ground.

What if I wanted to also know the firework that exploded both closest in distance, and closest in time to another particular firework? To know that, we would have to go back through our video footage of the fireworks display and record the time of each explosion as well. Now we have a 4-dimensional vector with 4 numbers: the three dimensional position of the firework explosion and the time of the explosion. Now we have a more descriptive type of embedding for the firework display by adding another dimension to our vector embeddings.

How does this translate to machine learning? Well, long story short, by processing a huge amount of text data, computer scientists managed to create embedding models that can transform a piece of text like a phrase, sentence, paragraph, or even a page, and turn it into a very long series of numbers that represent a point in a theoretical high dimension space.

Instead of 4 numbers, there are 300, or 700, or even 1500. These represent 1500 ways in which one piece of text can be “close” or “far” away from another, or 1500 dimensions of meaning. It’s a captivating concept for many that we have the means to create numbers that represent in some way the semantic meaning of a piece of text.

Using math, two of these high-dimension text vector embeddings can be compared to find out how similar or “close” they are to one another if they were created by the same model.

That’s the first thing that happens in this app. The user must put in a document and name it, and then choose a type of embedding. The server takes that document, breaks it into text chunks, and then turns each of those chunks into a vector embedding, and that chunk is saved as a row in a dedicated table for that document. Each document is stored in its own dedicated table to allow for the variable length of the vector embeddings created by different text embedding models.

Once a document is stored in the database as vector embeddings, the user can enter a query to “ask” the document. The query is used in two ways. The first way is to search the document. We don’t do a traditional text search, instead we are doing a “vector search”. The app takes the query, turns it into a vector embedding, and then finds the sections of the document with embeddings that are most similar to the query vector embedding. A similarity score between 0 and 1 is then generated for every document section, and several sections are retrieved from the vector database based on the top_k_similarity and the similarity_threshold. Basically, you can ask it how many document sections to retrieve, and how similar they must be to your query to qualify for retrieval.

That’s the Retrieval in Retrieval Augmented Generation. The next step is the generation.

Once computer scientists figured out how to convert text to semantically significant numeric vector embeddings, the next step was to create models that could produce text. They did so with great success, and now we have Large Language Models like GPT-4, LLama3, and Claude 3.5. These LLMs can take a prompt, or query, and deliver a completion, or answer, which is the text it thinks most likely to continue from the text presented, the prompt.

LLMs must be trained on large amounts of text data, and their responses, or completions, are limited to that training data. When we want the LLMs to provide completions that might include data not in their training sets, or ground their completions in a particular set of knowledge, one way to do that is to include extra contextual data in the prompt. Basically, if we want an answer from an LLM about something it wasn’t trained on, we have to give it the information in the prompt.

Many people found themselves in a situation in which they wished chatGPT or their local LLama installation could provide answers based on their own personal documents. It’s simple enough to search your documents for that information, paste it into the prompt, and put in your question, and people found themselves doing it manually. That is its own form of Retrieval Augmented Generation. RAG is just the automation of finding information relevant to the user query and providing it with the query to the LLM for a more accurate or useful response.

In this app, the document sections we retrieve with the vector search are sent with the query to the chosen LLM, labeled in the interface as the Model, to provide the context to the answer.

In the video example I made for this project, I ask the question “Who is the villain in this play?” with the documents “Hamlet” and “King Lear”, which contain the entire text of the two Shakespeare plays. The IRIS database already has two tables, one for Hamlet, and the other for King Lear. Each table is filled with rows of vector embeddings created from splitting the text of each play into sections. These embeddings are long series of numbers representing the many dimensions of meaning in each of the document sections.

The server converts the question “Who is the villain in this play” into a numeric vector using the same text-to-vector model that generated the vector embeddings for King Lear, and finds the sections in the King Lear table that are most similar to it. These are probably sections that mention the word villain, yes, but possibly other villainous things, such as treachery, betrayal, and deceit, even if villainy is not explicitly mentioned. These document sections are added to the query and sent together as a prompt to an LLM which then answers the question based on the provided document sections.

This is done separately for each document, and this is why the answer to the query is different depending on the document being queried. This completes the acronym, since we are Augmenting the Generation of our answer from the LLM with the Retrieval of relevant context information using the power of vector search.

Many thanks to anyone who takes the time to read this and I would be happy to expand on any of these topics in a future article. Feedback is always welcome.

2 comentarios

Comentarios (2)1

Inicie sesión o regístrese para continuar

【週間ダイジェスト】 7/22 ～ 7/28 の開発者コミュニティへの投稿

SQLでレコードをSelectするとSQLエラー -114 が発生します

Using Django and Vue.js to create a web application on IRIS: Peeking into the Django framework

Parse strings in any XML format

An Expanded Explanation of RAG, Vector Search, and how it is implemented on IRIS in the IRIS RAG App